Wow - I like the solenoid setup in my 1980 era 8274 (started as a 24V winch):
Essentially the same function as the OP's solenoids, just looks neater

Ignore the wiring comments, since they are for reconfiguring the solenoids for 12V operation versus 24V. Both the 12V and 24V winches use the 12V solenoids, just that the control wires are in series for 24V (12V+12V) versus in parallel for 12V.
Anyhow, the way the solenoids work is that solenoids 1-4 activate to go one direction and 2-3 activate to go the other direction. The fact you hear a click is one solenoid is activating (would be the top left (my #1) and your bottom right (my #4) isn't. So, either a dead solenoid or the control wire to activate the solenoid is faulty. A multimeter is your friend...
The 4 solenoids basically allow the + and - to the winch motor to be flipped +/- or -/+ to provide the direction change. It's pretty basic.
Albright is a good solution if your solenoid is dead. If just a control wire issue, then it's just a few cents to fix and you can decide if you want to go the albright path as an 'upgrade'.
cheers,
george.