Ok, I think I can now say I have it licked. There were three main factors:
1) DTI paralax/sine error. Thanks all for pointing it out. Moved to a plunger that I could align nice and square, and found a 15% or so error.
2) The edge of my cross-slide carriage has a couple of thou of slope. When I changed my setup to a block in front of a block, 3 thou of error difference went away. Next time I get a chance I'll square up that front edge somewhat better to avoid repeating the error caused by the belief that that edge was square vertically.
3) The real kicker: Induction from the power cables. I did a bit more number noodling, and a 60HZ interference with the edges being carried in the read head wiring could *really* mess up the signal, particularly when the frequency of the signal was near the frequency of the current. So yeah, when dialing in close it was becoming grossly inaccurate.
Anyhow, sorry for the long-delayed report. A new (to me) 40 year old non-working synthesizer landed in my lap late last week, and I've been trying to revive it. I've tracked down the not-functioning chip - the panel driver that controls the LEDs and the buttons - and desoldred it and replaced it with a ribbon cable and an Arduino. And tonight I finally got a first set of comprehensible signals to the Arduino. I've now reverse engineered enough of the thing's software (it's got a z80 CPU, so dead easy) to replicate the panel driver's functionality in the Arduino and with luck tomorrow I'll get to generate some sounds!
I mean, totally not the right venue for this project, but I'm very excited by it. I mean, this Yamaha PSR-70 was a cheap(ish) piece of trash in its heyday, and a replacement is now less than $200. But it could totally work again, for less than $200 of parts and 60-80 hours of my time! What's not to love!?