Sounds almost like my issue that was definitely starter. However, my starter has been replaced with a reman'd unit a number of years ago, got approximately 40k miles on the reman'd before it started having issues. Biggest thing for mine was the fact that I determined it was very much temperature related. When we would get cold snaps during the evening (getting blew about 40*) the next morning I would have issues with the starter. Above 40* or close to that during the night would yield little to no issues with the starter. It would only ever intermittently give me issues when warm.
New starter definitely has a different sound to it when kicking over, but I do feel like it was definitely the starter. I have the old one (paid to keep it) as I had a shop take care of the job for me, and will dig into it to see if the contacts were actually worn or not. Seems like that majority of the last 5-10yrs of my cruisers life was a driveway ornament so that could play into why the contacts were giving grief. If nothing else, I will have a rebuilt spare starter should the new reman'd ever decide to go out.