I don't quite follow why you could not use your jaws orientated to the smaller diameter with a parallel or sacrificial backstop material. I see the way you have it you get the benefit of the jaws as a back stop, but don't you risk drilling into it? (assuming its a through hole)?
This is why I like chuck depth stops. I just wish they were not so expensive.
Now if I had a 3D printer and some way to make them flat to say a half thou......