If you can feel the detents when you shift, then the ball and spring are likely OK. With the linkage disconnected, you should be able to manually move the shifter on the transfer case. It should "snap" securely into each position with some effort.
If the detents check good, then something is...
That's not how it works. There is a detent ball/spring in the transfer case that is supposed to keep you in the gear selected. The lever/linkage should move with no resistance. If you pull the cotter pin and disconnect the linkage to the transfer case, the lever should flop around easily.