I think that your Xterra is a fine choice for a vehicle if you are interested in a balance of driving on the paved roads or on the unpaved roads. If you want to get serious about being off road you may be disappointed. The opinion that I have developed is from what we see off road (defined as mountain trails that have not seen a road grader). We regularly see Jeep Wranglers, Cherokees, old Broncos, Scouts, Toyota trucks, Land Cruisers, Nissan pickup, even Range Rovers. I can not remember ever seeing an Xterra, once in a while we will see a Blazer or a full size pickup that someone is beating to death. I would hate to see you buy a truck that has limited availability of upgrade parts and be stuck with the wrong vehicle for what you want to do. When you get to town, look us up, come on a trail run and see for yourself what is on the road, what is off road, and what works of others. Then you could make a well informed decision.
Good luck in which ever direction you go.
David M