You can just leave the remote part inside the vehicle. If the remote is in the inside - it doesn't allow the doors to be unlocked by grabbing the handles. The car knows if the remote is inside or outside. If there is no remote on the outside - the passive keyless entry by pulling on the handle doesn't work.
Now if you're worried that some asshat will break the glass, get into the car, and drive away using the remote that's inside...
You could get a RFID blocking bag, put the remote inside, and lock the bag in the glovebox.
Here's a RFID blocking bag that I use for other things and know it works to block the remote's signal. Many that claim to - don't.
I had to try a few before getting one that worked.
Amazon.com: Ska Direct 100% Anti-tracking Anti-spying GPS Rfid Signal Blocker Pouch Case Bag Handset Function Bag for Cell Phone Privacy Protection and Car Key FOB - (Black): Cell Phones & Accessories
But one could easily break into the glove box. It's just plastic. And a thief who breaks into the car will likely rummage through the typical compartments. So maybe hide the bag and remote...
On model years 2010 and newer you can also disable the smart key system from the nav unit. When the smart key system is off you have to press the buttons on the remote for lock/unlock. To start the car you have to do the "dead battery in remote process" by holding the remote up to the start button. You'll hear a beep for confirmation and then the start button will work and start the car. When the smart key system is off it works much like a prox badge to get into a building to start the car.
Page 573-574 of this manual, for a 2010, shows how to enable/disable the smart key system.
https://www.toyota.com/t3Portal/document/om-s/OM60G31U/pdf/OM60G31U.pdf
So you could also disable the smart key system too. If one does break in and finds the remote they would have to know to hold it up to the button. Doesn't 100% prevent the issue but adds more of a challenge.
The most effective way to avoid a drive off is to not leave the remote in the car. But you're surfing and that makes it hard.
I see this as something a surfer might use. Another option is to disable the smart key system and then use this lockable hitch safe.
Assuming you're not towing at the time. Although it might stand out and be a target.
Amazon.com: HitchSafe HS7000 Key Vault: Automotive
The RFID blocking bag means you don't have to turn the smart key system on and off.
Just have to have a good place to secure/hide it.
Another good hiding place could be the jack compartment (there's cup built into the access door) or the storage compartment in the tailgate next to the tool box.
A visual of these are on page 516-519 of this manual.
https://www.toyota.com/t3Portal/document/om-s/OM60G31U/pdf/OM60G31U.pdf
Just some thoughts, ideas, and options...