I had similar problems and I would consider multiple things in addition to the panhards which I agree should be first thing ...
Things I did that have me tracking straight now (again, specific to my vehicle), in no particular order (except for getting panhards done first):
- Agreed first on NEW PANHARDS. It will help get the vehicle where it needs to be. When I did my first lift, a TJM and ICon at 4" I was having noticeable drift. Once I installed the new panhards, it corrected considerably.
- If you have caster plates installed on stock bushings then I don't see how that could be the problem unless you smashed the plates on a rock or something to throw them out.
- check steering and adjust as needed. Your steering wheel will likely never be 100% center but you can get fairly straight tracking. This doesn't really seem related to your particular issue, but is a suggestion anyway.
- check your tire pressure. If you are running stock tires, then it might not be as big an issue, but if you are running oversized 34-35, then you will want to run them higher. I currently have 35s and run about 38# and it seems to agree with the vehicle. When I go lower, I drift. If I go higher than 40# then it is a little squirrely and hard to control making sudden movements while driving - scared the piss out of me a few times.
- Get your alignment checked. You cannot adjust caster obviously, but adjusting toe can help considerably. After switching to a 3" full icon lift recently, I was having issues still with the above (squirrely) and the alignment wound up being a huge help. I got the alignment done and wheels rebalanced with weights instead of beads, at my last knuckle rebuild. Truck drives straight aside from the occasional minor sway from being about 6k pounds and lifted. But even with, it is easy to control and I have no issues.
Keep in mind also - once you touch this truck, it will never be the same, If you are particular about things then it will be a constant battel to get you where you want it. I'm fairly particular and it took me a while to put all the pieces together to get mine to where I can enjoy the drive.
good luck!