I have used a ‘full size’ (2 hp) knee mill for years. I apprenticed as a machinist with Ex-Cell-O Special Tools in the 1980s. We machined indexable carbide cartridges and custom boring heads for large milling centres.
Think about what you are going to machine and what type of tooling (Cutters, etc) you will be using. Tooling can make up about 1/2 your expenses when you get a good vise, boring bars, DROs and other metrology equipment . . . then you may want to jump into CNC to make use of you programming and automation interests.
I went the Desktop Manual route . . . for now.
My KING Mill, 110/1/60, 3/4hp is small enough to fit into my basement LAB, but can hog off mild steel with a 1” diameter flycutter. Sure I would like to have a full size knee mill . . . but not when you have to lug down some steps. 90% of what you are going to machine can be done on a desktop unit.
If your pieces are bigger, install a jib crane that can reach a good part of your shop.