If physically contacting the relay is causing it to switchover I'd think the contacts in the relay are sticking.
Looking at the drawing it shows the rh batter going to ground.
The lh battery goes to the relay. When the relay cycles it's sending the Lh battery to ground and this giving you...