I'm mostly thinking severe rear height might be pushing your rear end off, and perhaps a rear panhard rod would correct it. But it's my understanding that the shift should be straight and just offset (like a crab walk), not directional, so I'm not sure that's it. Which is why with my truck I was leaning towards slightly different CV angles due to a lean causing my torque steer. I actually put a 1" diff drop kit in to get the CVs back to their original angle, which did seem to reduce my torque steer further, though not completely eliminate it.I'll measure again today and see what the difference is side to side, but I believe it was only .5" and maybe 1-1.5" front to back. I hope you're right about the rear height being an issue, but I would assume when I tow it would correct since the rear sags making the rig level front to rear. But the only changes that directly corelate to when this issue became noticeable was the rear trim packers to give me some rake back and replacing my beat up coil isolators.. otherwise my mods had been there and there wasn't an issue.
When I had this similar issue on my daily, the root cause of that ended up being steering stabilizer bushings. I haven't seen anything on here about someone replacing KDSS bushings to fix an issue like this, but maybe that's a place to start?
Trailer makes it worse because weight applied to the rear (no matter the height) results in the front rising up. Go measure your front height before and after hitching (even with WD properly set) and you'll see what I mean. The extra 1/2" or 1" of lift you get up front when the trailer is hitched in the rear is effectively a taller lift, which means your CVs are at a steeper angle and all of your alignment specs (which were done when you were unhitched) are "wrong". It's like you aligned for a 2" lift but then swapped it for a 3" lift. So yes the trailer helps squat the rear, but the opposing impact on the steer axle is why you're squirrelly.
When the tech did my alignment he actually jacked the front of the vehicle up by the frame to similar a ~3/4" additional front "lift", as if I was towing. It's a marginal improvement at that point though.
Sway bar or other bushings could be worn and causing issues, for sure. My sway bar end links are tight but the bushings definitely have a bit of old rubber cracking going on. If you're not confident in the condition of your suspension components, I'd put it on a lift and check everything out. Worn tie rods for instance might lead to wandering and loose steering, though if it's only on hard acceleration and not also when cruising I'd be suspect in my own diagnosis there.