I totally hear you, but it really depends on what angle you're coming from.
Back when you bought your 40 brand new, you weren't thinking about it as a rare collectible vehicle, because it wasn't back then. The only reason to buy it was to take it off road, because that's what it was build for.
But most of them have since been beaten to heck and/or have rusted out and have been crushed. Now they're somewhat rare and appreciating assets.
I paid $11k for my rig, and I'm putting about $20k into the restoration (and thousands of hours). While I'm not doing a nut and bolt concours restoration, a large part of my motivation is to preserve a classic vehicle.
If I wanted a rock crawler and had the same amount of money to spend, I'd probably start with a TJ and modify it a lot. If it rolled on the trail, I wouldn't be heart broken... Mostly because it would have armor, and body damage could be fixed with a hammer and dolly and a rattle can. But I'm attracted to the 40 because it's a beautiful, nostalgic, well-built classic vehicle that I want to restore and preserve.
It sounds like you're attracted to the 40 because it can take you out into nature, because you can get your adrenaline going by taking some risks, etc. And I completely respect that.
But I don't think one approach is more valid than the other. It's an individual decision, and my decisions about my rig have no impact whatsoever on anybody else. I can understand and respect others points of view on this... But it's weird when people lash at about other people's choices when they have no impact on them... but perhaps that's a definining attribute of our times.