I've been chasing this for a couple of months so I can only imagine your frustration!
I've tried all the usual suspects, power is fine, fuel tank and pickup is clean as a whistle, I ran clear hose and there were no bubbles in the line, fuel filler cap breather is clear, cracking injectors doesn't isolate it to any particular cylinder.
I forgot to mention I replaced the fuel filter with a genuine Toyota one as well. The old one seemed ok but had some orange coloured flakes in it, although they looked more like sediment than rust.
I'm going run an alternate fuel source this weekend and try to rule out the fuel line. Will also park it on some steep hills nearby to see if it gets any worse which I believe will hint to a tired governor spring, I'm assuming you have tried this too?