There are sets of bushings and grommets on the ends of the linkage that connects the shift mechanism to the transmission, they wear and/or disintegrate. They are still available from Toyota.
90480-17177 grommet
90389-08005 bushing
90468-14017 Clip (sometimes breaks or gets lost, cheap...