The Fed is allowing its holdings of agency MBS to run off and many other big players in the MBS market appear to be close to tapped out.