I'm not so sure. There wouldn't be a load on the CVs, but they are still spinning. The front gears are rotating too so long as the CVs are attached. I don't feel like the noise changes based on the load (heavy or light foot, CDL locked or unlocked, etc). But the speed and volume of the noise do vary depending on my speed, and I'll hear it even when in neutral or braking down to about ~15 mph.
Either way I'll be able to rule out diff-related noises in a few weeks since the re-gear should cover all of those bearings. The big question is how much "while I'm in there" other stuff do I want to do. The driver's CV seems like it's worth rebooting (and inspecting) since it was rebooted about 45k miles ago, it's a non-OEM boot, and I actually ran into issues right after the reboot where it was slinging grease out and I had to pull the CV clamps (which weren't tight enough) and put on some worm-drive clamps. So it's possible the CV is actually low on grease and wearing oddly, or something. (OTOH I'd expect a bad CV to click, and mine doesn't, so WTF do I know?) Also my wife, my kids, and Rick at ChiTown all agree with me that though it's not entirely directional the sound does seems to come from the front, even after the tire rotation.
The passenger's CV was also rebooted, but it had a tear in the outer boot and Toyota did a re-reboot of that one at around the 90k mark. So while that *could* be an issue, honestly I have more confidence that one was done correctly...
If the gears and CV reboot don't resolve it then I'm confident it's either a bad rear bearing or my tires are absurdly loud without showing any significantly odd wear patterns.
All that said my theories have been wrong enough times that who the heck knows