wow. I can take .100 DOC in aluminum, until the motor stalls, but no chatter.
Put on the face plate and check axial and radial runout. Something isn't making sense to me.
Another check is to chuck up a piece of HRS, say 1.5" and put pressure on the bearings in all 4 directions and check for one direction being soft.
Lastly, check the seating of the headstock, and that the hold down bolts holding the headstock to the ways are very tight. 70% of yield tight. If they are at all not snug enough, you will get a lot of problems, including this one.
That's all I can think of right now.