Actually I found this flange issue to some degree on all three sets of bearings I worked on this summer, and found it to be only issue for N to D clunk. I'm sure other worn parts like spider joint, worn spline of propellers shaft yokes, gears in differential or transfer case can also be contributing, but I've not found that. Certainly bad CV bearings could also be cause of clunk. But as you've pointed out they will "click" while driving if bad.
Of the three I worked on, two of them had shops' service the wheel bearings replacing with new and replace front drive shaft recently. None had axle hub flange replaced. All had some degree of N to D clunk.
I've now replaced with new flanges or swapped. This has improved or stop N to D clunk. Most axle had some wear on teeth and so some backlash remained.
The Three 100's:
My 01LC 165K miles, purchased with 59K. Did my first wheel bearing service at 98K, found locking washer unbent/unlocked on one side and both sides loose. Both sides the claw washer scored, with the unlocked one being the worst. Didn't even look close at hub flanges until this summer, replaced when I did at 165K.
01LX 214K found wheel bearings loose and snap ring gaps excessive on both sides. It also had two new after market front drive shafts. Picture above of very worm teeth of flange is from this one. That was the worst wear of the hub's teeth of the three sets.
00LX 265K both side very loose, both locking as locking washers had all five tabs bent inward so bearing where walking off. Both snap ring gap very excusive one side more than other which flange outer surface (contact surface) for snap ring was toast.
After find these issue with backlash due to worn teeth, I visited with Toyota Dealers' head mechanic. He told me no amount of backlash is expectable on axle hub flange to axle teeth. When dealing with the N to D clunk on a 100, under warrant, they always replace both axle hub flange and front drive shaft with new OEM. He said if we don't we'll chase then cluck forever. He went on to say this is caused by improper wheel bearing service. In which bearing preload is set to loose and/or snap ring gap is to wide. This sets up a sawing action which cuts away at teeth. As bearing chatter (form looseness) scoring claw washer they loose further speed up the processes. This went hand and hand with what I've observed over the past 13 years. It's why you see me time and time again post here "set preload with a spring scale" even if torque too get there is uncomfortably high.
It's also why a check snap ring gap ever time, using a puller and clamp, this pulls axle out very tight. Snap ring gap can't be to tight.
Some "clunk" is to be expected with high millage. But pair that with loose wheel bearings and excessive snap ring gap. Well I've seen more than a few post over the years of failed axle hub flange in which teeth where toast (gone).
I've now come to the opinion; if the front drive shaft are replace for the extra $50 it's well worth using new axle hub flanges.
One can switch flange from side to side as suggested, but even better is switching axle from one side to the other and replace the flanges. This is something, after speaking with CVJ here in Denver, we came up with. CVJ felt some play (backlash) was ok or you'll not get flange on, I'm sorry but he is wrong. No backlash is best. If/when I go buy rebuilt front drive shafts, I'll take a new hubs with me to test for a tight fit.
Dealer's recently told me the outboard axle/tulip are no longer availed. It's in our best interest to keep our OEM axle in the best possible condition we can in our 100's, as one day OEM's will not be available.
Long response but this is one area of the 100's I've become very aware of. I'd like to see the massage get out on proper wheel bearing and front drive shaft service. It's my opinion we'd see fewer wheel bearing and fewer front drive shaft issues if shops' would take the time do correctly and cared for leaky boot timely.