I was in the middle of a very large decision, same as you, except it would be my first forte into 4WDs (at least in terms of ownership). It was a huge toss-up between a 3rd or 4th Gen 4Runner, an 80 Series, or the 100 Series I stumbled upon.
See the issue that I have with the 4Runner is, I'm rally not crazy about the body styling. 4th Gens are truly a hit or a miss (and still very expensive for what you get), and the 3rd Gens are too difficult to find in the condition I wanted; or, once you find it, it's already sold. The 80 Series, well, it's hit classic SUV status and everyone wants an arm and a leg for it (rightfully so, I still want one), or they won't give it up.
Now, as far as the 3rd Gen to 100 Series (or really just 4Runner to Land Cruiser) comparison goes, it's night and day... But at the same time they're both older 4Xs and have their quirks. You aren't winning a race in either, though the 3rd Gen would most likely gap a 100 in the 1/4 just because of the weight difference. With that being said, I daily drive my 100 40+ miles round trip every day, even more on weekends, and it doesn't phase me. You get used to it, and learn to stick to the right lane and budget a few extra minutes to get to where you're going. On top of that, you're getting all the luxury amenities the LC offers, and you don't have to go searching for a "Limited", they all come the same way (for the most part)...
Honestly, just yesterday I saw a pretty cool 3rd Gen and thought "yeah, I still really like those;" but the issue is I only like them when modified... and that is a lengthy process for someone like me. My 100 is basically stock, and I still love it, and can't wait to keep modifying it and daily driving it.
This is all personal preference -- and the novel I just wrote might even make the decision harder.
In short, life is too short to drive something you don't like every day -- I did it for two years (but now I can say I miss that car, as life goes). If you aren't happy, find something that fits you and your needs better, especially if the budget allows. I can say, and I know the awesome community here can as well, I absolutely do not regret my 100, even with all it's faults and issues. I know I can walk outside right now, fire it up, and drive (pretty much) anywhere without hesitation, with ease, and in comfort.
ahem.. /rant(?)