My tailstock has an MT3 Taper with the tang provisions so I am happy using it to drill with. It is also 4 inches, but not with a tang. The tang will self eject at 0.75 inches so I lose 3/4" of travel. It has never been an issue and even if it was, I love the auto eject feature so I'd simply live with it.
I have an MT3 tool holder that came with my lathe that I've never used. I will have to try it your way just to get a feel for your joy.
One of the concerns I would have with using a tool holder is concentricity to the spindle. It's easy to adjust side parallelism, but how do you adjust the vertical? Just pray that it's ok?