Correct, I've left it unplugged and everything is fine.
It mounts on 2 locating pins, so I'm sure this particular part is aligned correctly, and it came from a Toyota dealership as a new part. But as @kbfj621989 mentioned, other stuff might be misaligned. I'll take more apart.