Something to keep in mind besides a lack of sidewall height is the added unsprung weight that the 20's add.
Not only will your suspension be working harder on the street making your ride over the potholes suck, that added weight will compress your already reduced sidewall even faster when offroad obstacles are encountered.
Most vehicles I've driven with 20's suffered in the ride department due to the added unsprung weight.
I'd be willing to bet a 20" alloy wheel with a 35" tire combo is lighter than a 16"/35".
Depending on the tire/rim combo, what he said.
And weight isn't an issue. Lots of 80's overseas running steel rims and heavy tires without issue.
The reason 20" wheels and 35" tires suck is that the side wall is so short, they're typically made of a heavier side wall construction. Seen more than one tire that adds a ply when you get to that ratio. E.G.: 2 ply for 16/17/18 inch rim and 35 inch tire, 3 ply for 20 inch rim and 35 inch tire.
Less side wall means less room for the tire to flex and compress, which means more of the impacts from the road are transmitted to the vehicle.
You are right that a 20 inch rim and 35 inch wheel will have a worse ride than a 16 inch rim and a 35 inch tire, but it has nothing to do with weight and everything to do with the short sidewall and construction of the tire.
FWIW I'm running steel H1 rims with 37" military tires which are
far heavier than the alloy 16's and 35" tire I was running previously. The ride is actually improved partially because I run at a slightly lower PSI (30 vs 35), but the bulk of the improvement comes from the significantly larger sidewall allowing the tire more room to flex. (And this despite a much heavier construction, 4 ply vs 2 ply.)