So you're shaving the keyway in a couple of thou. on each stroke, cool. That's how green I am when it comes to the possibilities of what a mill can do.
Did you modify that boring bar yourself or is that something that can be purchased?
I found these guys but their tools look expensive.
https://cncbroachtools.com/
If you want a simple version of a broach to stick in your mill use 1/2 - 5/8" round bar, and 3/16" round carbide or HSS blank and a 4mm grub screw.
Cut the 5/8" bar to a length long enough to travel the length of keyway you need, plus enough for length for clearance, for projection into the collet, as well enough length to leave sufficient material below the cutter mounting point.
You need to then drill a 3/16" hole thru the middle of the bar, and a 3.3mm thruhole perpendicular to the first hole so it enters the bore of the first hole. Tap that second hole for M4 grub screw.
Then grind a flat on your blank to the midpoint of the blank at the front tip.
Then grind 5 degree primary relief angle.
Then grind a nice deep secondary relief angle.
Then grind a flat on the side as shown, perpendicular to the flat on the top. Finally cut off the cutter to just shy of 3/4" long.
Insert your cutter into the shank. Insert shank into the mill quill using R8 collet. Ready, set, broach.