1) The length of time you should expect from your paint/primer is inversely proportional to the amount of surface rust or other contamination left on the surface prior to coating. If you want it last, get the rest of the rust off.
2) No metal left that smooth is ready for primer. You need to create a surface profile by sanding or blasting if you want proper adhesion from any primer.
3) Here’s where I tie #1 and #2 together: wire wheeling and sanding won’t get all of the rust out of the pits, they just can’t access the recesses of the pit. Your best bet is to blast the fenders because it will remove the remaining rust and provide a great angular profile for optimum primer adhesion. The good thing is that you’ve already done the real hard work and all they need is a quick sweep blast which can be done with inexpensive home equipment.