Use carbide bits from pro machine tool shops, not introductory grade cobalt stuff from hardware/chain stores. About 5 minutes per side and paint matches when you do a patina resto. But to each his own
I hear you, have done so too. Buy some good quality bits, I like carbide for rusty stuff, although they break easily, they power through and are cheaper than getting new hinges.
For future reference, drill out the pins instead of destroying the hinge bracket by trying to bash the pin out. Same with early hood hinges (use a door pin, cut the round end off and bash the ends flat to lock into place for the OE look). If you do it right even patina paint survives.