I tried drilling them out but they just started spinning with the drill bit. I ended up using an angle grinder to wear away the exposed portion of the rivets. The inner part of the rivets fell into my doors but I haven't heard any rattling or anything.
Just be a bit careful with the angle grinder and use small movements so you don't accidentally grind your paint. The portion of the angle brackets that're flush with your side panels provide plenty of protection (and fun sparks) if you've got a steady hand.