Fannie Mae and Freddie Mac this week announced a relatively modest 3.4 percent increase in their conforming loan limits for 2004. The conforming limit for single-family mortgages will climb from $322,700 to $333,700 on Jan. 1, 2004. In terms of percentage growth, it marks the smallest annual adjustment since 1996, when the agency loan limit rose just 1.9 percent.