I've used 135, 118 and spotting drill/mill bits (of various angle configurations) in all sorts of sizes from small to large in both mill and drill press.
The important things to do are:
- good center punch when drilling manually.
- good initial alignment when starting.
- sharp bit (can't say enough here)
- correct drilling speed, if in doubt slow it down (particularly on the drill press). On the press I generally run slower for all bit sizes as it helps with alignment and at time of final break thru.
- locked in place helps for real precision.
- finally, practice, practice, practice....it does get easier as you start drilling lots of holes and develop a feel for machine, tools and material.
Ultimately for most its practice, but in production I do whats required for the best, most accurate, fastest (this is what makes money) and costs the least in tooling and set up result.