The Community Home Lenders Association believe the sharp increase in fees for second-home mortgages will impact middle-income families.