Fannie Mae and Freddie Mac have had a relatively challenging time trying to grow their mortgage investment portfolios over the past year, despite record volumes of new production. In its first quarter earnings statement released this week, Fannie Mae said that its retained mortgage portfolio had shrunk for the second consecutive quarter, at an annualized rate of 7.6 percent, during