Market size, interest rate risk and a novel asset — mortgage servicing rights — combine to create a complicated investment decision for banks and private equity.

Prosecutors say the defendants doctored the income and purchase prices multifamily properties by maintaining a second set of books, and using fake liens, false contracts, and inflated invoices...