Mortgage profitability gyrated wildly from quarter to quarter in the earnings reports for most lenders in 2004, but the overwhelming direction for the year as a whole was down sharply from the record levels of the 2003 refinance binge. A preliminary analysis by Inside Mortgage Profitability of fourth-quarter earnings releases shows virtually all companies reporting hefty declines in mortgage banking income on a year-to-year basis. With about half the industry having reported earnings as of…