I have made a few BXA tool holders quite successfully. The "trick" (if you can call it that) to getting the dovetail the right size is to measure it using two dowel pins that are big enough to be useable and small enough to fit against the two flats. Then measure their gap with a caliper. The measurement isn't that critical. A few thou one way or the other doesn't matter. I believe I used 3/8 shanks from broken end mills.
Get the groove depth right using conventional means. And also get the top width of the groove right using conventional means. Then start cutting the dovetail till its big enough to take the two dowel pins and measure the gap between the pins. Compare that to the same gap on a purchased tool holder. The difference is how much more you have to cut from both dovetails - 1/2 the difference for each.
That might sound complicated but it's really easy peasy. Ask questions if you need more info.
Btw, most of those holders also have a small step at the dovetail cut. Duplicate that too.
Get the groove depth right using conventional means. And also get the top width of the groove right using conventional means. Then start cutting the dovetail till its big enough to take the two dowel pins and measure the gap between the pins. Compare that to the same gap on a purchased tool holder. The difference is how much more you have to cut from both dovetails - 1/2 the difference for each.
That might sound complicated but it's really easy peasy. Ask questions if you need more info.
Btw, most of those holders also have a small step at the dovetail cut. Duplicate that too.