I was on the same boat when I purchased my '08 GX470. I love the 4runner, but the price on them out here in norther california is nuts. GX470's are starting to get attention so the price is going up, but they're not as over priced as 4runners in my opinion. With GX470 you do have to factor in cost of replacing the factory shocks and the rear airbag suspension at this age, but as everyone said you do get a V8 and overall it's a nicer interior to be in even when offroading. I mainly use them to go up to tahoe for skiing or some fire road trails for camping/fishing and nothing crazy. as a reference I paid $14k for my '08 with 125k miles on them. bit higher than I wanted but the car was in really good shape.
'04 4Runner will not be VVT-i if it's the 4.7 V8, that started in '05. So really if you want a LOT more headroom, more electronic stuff to go bad, and basically all the same bits as a limited V8 4Runner from '05 , the GX470 is a great pick. 4Runner would also have dial to go from 4hi to 4lo vs the transfer case shift lever and knob in the GX. GX will have more sound deadening and softer leather too. I also found that Lexus original owners are more likely to never take their vehicles off-road and always have them serviced at a Lexus dealership. We bought our 4Runner new, love it, would own another or GX without question. I almost got a GX when I decided to get another FZJ80 instead ( it has my heart)