How about some Velcro (with the adhesive backing) - just the 'hook' side which has the rather stiff hooks.
There's various thicknesses (like the industrial strength etc) Velcro. If you cut some strips of it and stuck it between the diffuser and the sides then it would 'fill' the gap somewhat and provide enough resistance to stop the diffuser flopping around but still have enough give to allow you to move it around. Shouldn't wear out since the hooks are very durable and relatively stiff.
This takes a bit of work, but works well. Remove the vent, then take the diffuser out. It pivots on a pair of pins that pop into holes in the vent housing. Find an o-ring that fits over the pin and pop it back together. The o-ring takes up the slop, makes some drag and lasts a long time.