Yes, that is troubling. Why i want to see if it is at all possible to shift without the usual clutch maneuver.
Any possibility you installed the wrong clutch fork pivot?
I have a tiny bit of experience here. My 12/'74 FJ55 came the the 1.5F naturally, a 4 speed tranny and 3 speed flywheel and clutch.
So, yes you can (imnshe) use 3 speed clutch components with a 4 speed transmission. But, you must use every single component for it to work. The clutch fork...
Have you tried double-clutching on downshifts? Shouldn't have to but maybe it'll help. Have you learned how to shift (up and down) without a clutch? Can you do that?
I'm hoping it's just a clutch adjustment issue too...