Tailstock alignment is for sure the first thing to check. Lots has been written on the forum if you do a search. Misalignment will cause all kinds of grief with any drilling operation but also will introduce taper into tailstock supported parts in basic turning. Depending on the lathe it can drift over time or get bumped out of position
It is better to center drill with an angle equal or larger than the drill angle. So if you have the typical 118-deg drill set, better to start with a 120-deg center drill. Lots of folks use the classic 60-deg pilot drills, but this lateral displacement (kick) effect can result when the drill grabs the crown edge & kind of starts self feeding in that direction. Also if your drills are not ground quite accurate, problems can ensue even on a perfect center drill hole.