Housing affordability concerns are being blown out of proportion, according to Stanley Middleman, CEO of Freedom Mortgage.