Welp...
I figured it out. The hard way of course because that's just how I roll.
I took it all apart, convinced that I was going to find a crack in the intake manifold. Also since I had the original carb from the '78 engine I thought maybe I'd just put the '78 intake on with that.
However I quickly discovered that the 78 intake does not mate up with the 73 exhaust manifold, I only have a long tube header from the '78 on the exhaust side (someone tuned this thing up at some point), and every single part from the 78 is rusted and nasty, which is why I put the manifolds from the '73 on this on the first place.
However on taking it apart I realized that the brake booster line goes into the intake arm that feeds cylinders 5 and 6, so then got myself convinced that the brake booster was causing a vacuum leak, which was my problem.
After screwing around for an hour or so with the carbs and realizing I'd need to completely rebuild the accelerator pedal linkage as well as the fuel line (I ran a really nifty, super custom, high precision hard line from the fuel pump to the carb), I decided to put it back the way it was, but better this time.
So I used permatex red on the mating surfaces of the intake and exhaust manifolds and gasket, and put it all together again. Let it cure, and plugged the brake booster line (goodbye power brakes).
Took it for a test drive and it ran absolutely perfectly, sounding smoother at start and staying smooth after it warmed up! Suspicions confirmed, it was a bad brake booster!
So just to be 100% sure that I was 100% right (again), I hooked the brake booster back up to make sure it would run like s*** again with that massive vacuum leak it would introduce and... it didn't. Still runs like glass.
It was the gasket. And I'm officially a slightly more experienced dumbass today than I was at the start of this debacle.
Since it took nearly the top half of the motor apart again though, I did double check the clearance on all the valves and confirmed that here all dead on spec. Like, dead on. Someone did a lot of work to this poor old rust bucket before it donated it's drivetrain to my project, w.for which I am incredibly grateful
Thanks as always for everyone who followed, read, laughed at, and chimed in to this thread with excellent advice and insights!