Refinance mortgages accounted for 80.1 percent of agency production back in 2012, but that declined to 65.6 percent last year and just 45.3 percent of fourth-quarter business.