Razor blade and some type of mild abrasive - whether on a drill, dremel, or air tools. Can be discs, scotch pads, or even a wire wheel. I like this method because if you plan to use silicone/RTV again, it cleans AND preps the surface for good adherence with the RTV.
I haven't found anything faster (solvents included) that is faster and more effective.
I haven't found anything faster (solvents included) that is faster and more effective.