I installed my sound system back in March and I had the same dilemma on whether or not I should remove the factory radio. It was a huge headache because I realized that the factory radio, head unit, and navigation are all tied together in grounds. I ended up dropping the project and left factory radio in place and just used the wires exiting the factory radio into a LOC then to an aftermarket amp.
The only thing I can really say I can't stand is the hissing sound the speakers have when you turn on the head unit. Did you have this issue? or rather do you hear this now with your factory amp back in place? OP
Here was my original thread if you want to take a look.
2005 Toyota LC Audio System Upgrade