'82 Door Handle Mechanism rubs door panel from inside

[gmac]

I have a 1982 FJ45 Troopy, and recently replaced both, the door latch mechanism and the door handles.
On both sides the door handle mechanism rubs against the door (right where the door handle bolts to the latch mechanism, to the point where the door handle does not go back down on it own (and you can see the door panel move out from the outside), but you need to push it down every time you open the doors.
Everything looks to be mounted like it should, and like it was before I replaced them. I can see evidence that this was happening before as well.
Below are a couple of photos, I circled the area that hits the door panel from the inside. Any thoughts?
Thanks
Guy

 

[gmac]

Has anyone run into this in the past. Could it be my door is bent? It surely does not look like it is.
Any thought from anyone?
Thanks
Guy

 

[AdamKFarmer]

Did you replace the latch? I had this problem as well with a new latch I bent that little ear that the handle connects to to clear the door.

 

[gmac]

Thanks Adam. That's probably what I will end up doing. I was surprised that the old latches did it and the new ones do it as well. Both doors. Will bent the ear and hopefully that will fix it.
Thanks