OMG, haha, are we going to open up that can of worms again about headstock alignment, tail stock alignment, carriage and crosslide alignmentMy tailstock has an MT3 Taper with the tang provisions so I am happy using it to drill with. It is also 4 inches, but not with a tang. The tang will self eject at 0.75 inches so I lose 3/4" of travel. It has never been an issue and even if it was, I love the auto eject feature so I'd simply live with it.
I have an MT3 tool holder that came with my lathe that I've never used. I will have to try it your way just to get a feel for your joy.
One of the concerns I would have with using a tool holder is concentricity to the spindle. It's easy to adjust side parallelism, but how do you adjust the vertical? Just pray that it's ok?