After working for more than two years toward a spinoff of Newrez, Rithm Capital paused the effort and will invest in the mortgage unit.