I've had both a '96 Hilux Surf w/ 1KZ-TE and 2X HDJ81's, both have their merits.
The Surf was great around town, smaller size makes traffic and parking lots a breeze, tight turning radius, that amazing roll down rear window, multimode TC gives you 2wd, AWD, and 4x4 w/ low range options. It was a great vehicle. The engines have suffered overheating failures, though no where near to the same extent as the 2.4's did. Make sure to keep the cooling system in top shape, add an EGT and coolant temp gauge you can trust, and it should be fine. Weak points are the same as the 4Runners of the same era, change the lower ball joints and either replace the rad, or bypass the factory trans cooler in favor of an aftermarket external cooler to avoid the "pink milkshake" should the rad fail and mix coolant and trans fluids.
The HDJ81's are bigger in every way, and better in many ways, but they are not as nimble, and sadly no roll down rear window (can you tell I loved this feature?).
I got way more looks in my Surf than I did in either of my HDJ's, I liked them both, but there was no question when I lost the 1st HDJ what I'd be getting to replace it.