Drop in GSE Loan Limit Will Harm Special Category QMs, Industry Says. In a letter late last week to the Federal Housing Finance Agency, the regulator of Fannie Mae, Freddie Mac and the Federal Home Loan Banks, the Mortgage Bankers Association warned against any lowering of the conforming loan limit for the size of mortgages that Fannie and Freddie can purchase. “Any FHFA action to lower loan limits would ... undermine the special category of QMs created by the CFPB for loans that are eligible for Fannie Mae and Freddie Mac purchase,” the MBA...