Loan officers in a recent survey said the lack of starter homes, and not high interest rates, was affecting housing affordability.