Not sure if this is the part you are talking about - this is the one that fit my 2006 LX. I posted this about a year ago. I had tried epoxy, then looked at 3d printing a part - before I realized the easy solution was to use the Camry part. Cost was about $25 last year.
For anyone who is interested - I purchased this part from ebay:
2002 - 2006 Toyota CAMRY Lexus ES300 Console Armrest LATCH SET black upper latch, and it fit my 2006 LX perfectly. The black color was a perfect match - it is the same part.
You will need to file away some material to make it easer to get to the screws. I used a Dremel with a small router bit and removed enough to allow me to take the screws completely out of the assembly, and allow me to get a screwdriver in to tighten them back up.
This is for the top latch assembly - the blue box in the earlier post. (or the part on the right of my picture.)