GSE Portfolio Growth Far From Certain in 2004 Interest rates are creeping up and Fannie Mae and Freddie Mac are sitting on piles of capital, but industry observers are starting to believe other market forces – combined with regulatory risks – will keep the GSEs from growing their portfolios significantly in 2004.