More banks are considering leaving the residential lending/servicing sector or scaling back. Will these decisions, made during tough times, come back to haunt them?