This is what I need to make
I'd rough cut it to length on the band saw with say 1/4" extra. Then machine the back of the large diameter flange first. The only dimension that really matters in this op is the flat rear surface. The OD can be cut too, but I prolly wouldn't because of the stick out. Gronk on the Jaws. Don't worry about marring your aluminium bar. That will get machined off later. This will be a facing operation only for the backside of your part at the big flange. Then I'd remove it from the lathe and drill and tap sacrificial flange holes from the back side small enough to be able to be drilled out later to the final required hole size. Next, I'd make a steel arbour out of 1/2" flat plate and mill flats in it for the lathe Jaws to hold solidly. The plate then needs to be mounted in the Jaws so the plate sits proud of the Jaws so it can be faced flat and true to the spindle. It also needs a center hole bigger than your final part ID. Mark the plate and jaws so the plate can be reinstalled later in the same place. Then remove it and drill it for the same bolt pattern as your part. Maybe with recessed countersinking so the screw heads don't hit the lathe jaws.
Now you can bolt the two parts together solidly with a flat 1/8" sacrificial aluminium "gasket" between them. If you want, you could also pin them - your call on whether the extra hole matters. The pin holes can be drilled as an assembly because the objective is a solid one piece assembly. All the bolts and pins must be short enough so they don't interfere with cutting the inside flange face later.
The sacrificial aluminium 1/8" plate between the two parts will let you cut the flange OD later without worrying about hitting the steel back plate.
Mount this Assembly back on the lathe with the steel plate in the same location as it was before.
You already know the rest.
You could optionally cut a big steel bar to do the same thing but act like a mandrel. But I'd use a plate myself. In either case, the key to all this is two flat surfaces (one in the steel plate and one on the back of your part) that are both perpendicular to the lathe axis. Once you dial those flats in, the rest will naturally machine concentricly to the spindle even if it wobbles to begin with. Just get those flats square.
You can drill a center in the long part to use while you machine the external surfaces, but I don't think it's necessary.
If you later find that the rear face is not concentricly square, you can always dial the part in on a 4 jaw with jaw pads and touch up the face.
The key to all this is a steel mandrel plate strong enough to hold your part for all the required operations without messing up your aluminium part getting clamped in the chuck jaws.
Machining the through hole will be a trick cuz of the length. But that will be what it will be. I'd drill it first, then use the biggest boring bar that fits tto machine the small end - which is a critical dimension if I understood correctly.
Broaching the bore will be what it is. I don't envy you that broaching job.
I'll think about it a bit more and see if I can come up with a better method, but I'm happy with this approach.