Tighter margins and a seasonal slowdown in home buying are prompting lenders to consider an exit in the coming months.