What tire pressure are you running? And what tires? My guess is the problem has always been there but was masked by a good steering stabilizer and probably good shocks. Now that those are getting some wear, it's showing up.
Without playing with the caster I don't think there is anyway to stop the wandering. Mine was quite bad and when you hit the brakes it grabbed ruts, swayed violently each direction etc. I had never addressed the caster problems (I have OME caster correction bushings, but more lift than they can correct), and like your's, it kept getting worse. I think for me it's a mixture of dead steering stabilizer (mine has a dent in it, is your's still the factory one?), loose shocks, etc. I did a cut/weld mod to adjust the caster and honestly it made a huge difference, rather surprising actually.
But check tire pressures first, also rotate your tires, etc, see if it makes any difference at all.