Hard is a very subjective term. This is not a weekend job. I had an entire BJ60 to swap everything over. We took the entire wiring harness, as there are a lot of relays and connections that are different between the FJ and BJ (Dash with glow light, water separator, glow relays etc). The motor mounts are different and in different places, so you need to cut the motor mounts off the frame and weld new ones on. The transmission crossmember is in a different place and had to be moved (more cutting and welding.
I swapped to 4.11s and I have 33x10.5 tires, and I actually miss the 3.73s. I think the 3B was happier at lower RPM on the highway.
I agree with
@Lead Head, a manual transmission swap for the 3FE would be a far easier swap than the entire drivetrain. the B series input shafts on the transmissions are different than the F&H, so you will have to find the right 5 speed for your 3FE.
I see you are in vancouver, do you plan on going anywhere that is cold? the 3B doesn't make much heat when its idling and putting around in traffic. I dove mine through a winter and calgary and damn near froze to death.