The click is the solenoid pulling the plunger down and engaging the bendix. If you have to hit it to make it even click, it points to either a bad connection inside the starter, (ground, main cable, etc) or a gummed up plunger. Regardless of what it is, you need to pull the starter and take it apart. The contats are about 8 bucks from kragen and a known weak point, so you might as well put new ones in.
FYI the flow goes something like this: You have constant voltage at your solenoid via the main starter cable. When you turn the key, you provide power to an electromagnet in the solenoid which pulls the plunger toward down. This pulling motion also engages the bendix drive via a shift fork, meshing the bendix gear with the flywheel. When the plunger comes all the way down, a brass ring on the underside of the top flange, seats on two L shaped brass contacts, and sends the constant power from the main starter cable into the starter motor where it spins the armature and goes to ground.
So, If it won't turn, the problem is either contacts, a short in the field coil, or worn out/dirty brushes.
If it won't click, that means the solenoid is not engaging, so you either have a bad solenoid, a dirty solenoid, or a signal voltage problem.
Since your problem can be addressed by hitting the starter, I'd say that you probly have a gummed up solenoid and/or a dirty signal connection from your ignition switch.
So, clean the connections and if that doesn't help, pull it apart, replace the contacts, clean/replace the plunger, bench test and reinstall. total cost, about 10 bucks.