Dwindling refis and record levels of home equity are making home-equity products attractive for nonbanks. The challenge is competing with banks that have the capacity to hold HELOCs and the like in portfolio.