I just removed mine last night. 2003 LX470. This may not be 100% accurate as I'm doing it from memory, but it should be about right.
1. Remove cup holders from back of center console. Open the cup holder drawer, manually fold the cup holders up and look inside. You will see two screws. Remove those and the cup holders should slide out.
EDITED TO ADD - pry back trim off of center console. Then disconnect two connectors (I think attached to the rear audio controls)
2. Remove 2 10mm bolts from inside center console.
3. Remove 3 (I think) phillips screws from each side of the center console. They are hard to get to, but move the seat around to gain access to each one. I used a right angle screw driver.
4. Pull the console up and out.
I believe that should do it. You might not be able to remove the plug for the changer without pulling up the shifter console (which the front cup holders are attached to).
This is what you are left with. 4 10mm holding it down.