“If we had a major economic downturn, I don’t know what kind of havoc we would have right now, because there’s no stability,” said Ted Tozer, a former president of Ginnie Mae.