Some axial play is normal. The output shaft is a splined input and held captive in the rear cone with a snap ring. The radial ball bearing is also held captive with a snap ring. So it's a lil sloppy.I just pulled off the rear drive shaft, and I finally found something.
The rear TC output flange has a good amount of play in it; I can move it in and out maybe about +/- 2mm. So I am pretty sure that's the culprit. Not sure why I missed this before, probably my fault. I am a bit embarrassed that I didn't catch this earlier...
So I could do a couple things to get this fixed:
1. Order a new bearing, pull off the rear of the case, replace the bearing.
2. Find a used transfer case, swap in the rear housing (then I get a good used bearing and get the VC back)
EDIT: There's a couple 200k TCs on eBay for ~$350 shipped
3. Order a new bearing, find a good used VC, R&R
Let me know what you guys think. Thank you everyone for the help on this, I think we finally got it!