The problem is that the reamer will follow the hole's least resistance. The damage certainly isn't symmetric, and the large flakes that will come off will tend to divert the reamer. It doesn't take much to get it off center (or worse ream an oblong hole - I speak from experience here).
It is surprisingly easy to chuck it in a 4 jaw chuck, set up the taper, and take a skim cut with a boring bar. This will take out all the random scratches and potentially hard spots from compressing the chips into the bore, and make reaming the way you suggest a dream.
Another tip: when you chuck the reamer and ream, don't do it under power! doing it by hand is far more controlled, and you can 'feel' the reamer work. Don't take more than 2 turns before removing, cleaning the reamer and bore, and then continuing... That way your reamer will last for the next time you need it.
I have a MT3 reamer that has restored 5 or 6 tailstocks, and it is still very sharp.
BTW, if your boring is off a fraction of a degree, it doesn't matter, the reamer will restore the correct angle. If you are off, say a half degree on a MT3, it is out very little over the length, which is easily corrected by the reamer. The .020 - .030 depth of scratches is far too much to ream alone.