What I described was how mine sounded when I first completed my swap, but my "rattle" sound stopped when the clutch pedal was pushed in. Mine turned out to be be a defective pilot bearing.
I also ended up adjusting my TO bearing so it just barely keeps in contact with the PP so it spins with no load all the time. My clutch set up was the old style that is supposed to be set up with a small gap. After many discussions on mud and 4BTswaps, I tried it and after 40k miles, no issues.
Mu rattle sound was also discussed on both sites, and several times the torsional vibrations of a 4 cylinder diesel was mentioned. This type of vibration can make the trans shafts rattle some. Isuzu uses special seals to help with this per Dougal in NZ.
I don't if you have had a chance to drive it enough to notice if the rattle gets louder when it warm? I found the cold gear oil made it quieter, and mine gets a little louder after a long drive.
How much flywheel weight are you running? My new engine will be around 62 pounds. The "Guru's" all have agreed that more is better, and should help smooth the 4's vibrations.
Keep the your head up and march forward, the end is near!