Depending upon how bad the wobble, yes I'd worry about it. It will wear out the fan/motor bearings sooner if not addressed.
Are the fan blades clean? Is there anything at all stuck to any of the fan blades? Or a chip out of the any of the blades?
And just in case you need more help:
v-belt replacement - ClubLexus - Lexus Forum Discussion - https://www.clublexus.com/forums/gx-2nd-gen-2010-present/909871-v-belt-replacement.html