Dang, that's some carnage, but I think it looks promising. At least it's giving way!
Did you drill out the rubber or chisel?
I'm hoping the freezer trick works on this thing. Would definitely suggest applying heat to the housing to increase the clearance. But be sure to smooth out any burrs introduced by the removal process. Maybe even a little grease on the cold side. Tap in with a gentle deadblow or rubber mallet.
It took me several hours longer than it should've to replace my u-joints last weekend because the yokes are angled and rounded on the outside faces. So the usual tool didn't work so well and things kept getting misaligned. My point being that I'm not sure Toyota always hit the nail on the head when it comes to maintenance access. Well, that is for people who can't afford each and every SST called out in the FSM!