My guess.....(did not read the entire thread):
Combo of
1) sticking egr valve. Allows it to pass the factory manual egr test, yet doesn't close all the way. can be Intermittent. Can be sometime corrected by tapping the egr valve with a screw driver handle. You will hear a clean up of the idle sound. Solution: short it out. Have heard that you can tear it down out of the rig, wire brush the carbon, and reinstall. Watch diaphragm though.
2) Cracked (could be small) intake manifold due to old sticky spring activated exaust by-pass valve. Solution: could weld it, but that is sketchy due to intake/exaust reassembly alignment issues. Best bet w. least risk (probably same cost as weld) is buy brandy new exhaust/intake manifold assy (say $600 or so).
Don't ask me how I know............