Market size, interest rate risk and a novel asset — mortgage servicing rights — combine to create a complicated investment decision for banks and private equity.