We traded in a perfectly fine 2017 base model (30K miles - Brandywine Mica exterior, black interior) for a 2020 HE (Blizzard Pearl exterior, black interior - no choice). We have no regrets.
Concerning terra interiors: I was indifferent, but the wife vetoed terra. She simply didn't like the way Toyota employed black parts (seat backs, some door panel parts) with the Terra. I wonder if her opinion was formed by our older LX that had a light interior but seat backs matched in color, etc.
The HE wheels are BBS forged and lighter than the base model wheels (in the same ballpark as the Rock Warriors). I am no longer a serious off-road type so the advantage of 17" Rock Warriors over 18" HE wheels is lost on me. Your use might sway you differently. The cleaner rear interior (if the base model seats are removed) and the lack of the cool box were both attractions to us. The HE has a cargo curtain that we never use that's not available on the base model (mall crawler - hide the packages thing?). The cool box was something we only used maybe twice. Anytime we were carrying anything we wanted to keep cool, we ended up with a small cooler anyway (drinks before and after trail, sandwiches, etc.) Maybe ours wasn't insulated correctly, but the base model cool box we had would not keep things cold long enough to have cool drinks or food there after a couple hour hike.