yellow connector

Hi, I have an amusing problem. When I unlock my door from the driver's side door handle (using physical key, not fob buttons), (i) the door does not unlock and (ii) the alarm goes off. Of course, if I leave before people wake up and forget to unlock from the passenger side, I wake people up...