Clr wont hurt the paint. The Brillo pad will scratch paint eventually.
Over time an oxidation film is developed on the paint that fads the color. Light scrubbing with clr/Brillo will remove and bring back a brighter color. Scrubbing is a balance of removing rust /paint-oxidation and not damaging paint.
Photo of before/after with clr&brillow on tire carrier comparison:
View attachment 2465153View attachment 2465154
I like a little rust, so I left some on. I could have kept scrubbing and removed more. If you want to remove 100% of rust, I would still recommend clr&brillow with touch up of phosphoric acid (navel jelly).
To prevent rust, people use fluid film or woolwax based coatings (check amazon). Apply once a year. Rubberized coatings like bedliner are notorious for rusting out, so I recommend staying away.