Part of it could be springback in the material, thats normal. The part is held rigidly on one end (chuck jaws) but not supported on the other. The cutting tool is pushing against the part so even if the tool is considered 100% rigid, the part flexes. Analogy is a person standing on the end of a diving board, lots of deflection under the load vs back where its anchored. So another pass at the same DOC setting explains why you typically see fine swarf as the material is slowly relieving itself, coming back to its axial orientation. Skinny boring bars see this too, now you 2 things, flex in the part and flex in the tool. Threads are kind of a special case because you are cutting the face of the thread. Even on the finishing passes shallow DOC the tool is contacting a relatively large oblique area on the of the tool vs any kind of single point cutter. But whatever you are getting on a free machining rigid material like brass would probably be worse on a more flexible material especially if it has more un-equal properties like resins.
A big factor that is largely hidden to hobby machinists is how repeatable the actual DOC setting is. When you make a cutting pass, typically you disengage the cross feed, get back into position, reset the DOC to the exact same position on the dial & re-cut. So you had it set to 0.054" DOC & reset it to 0.054". But is the tool really at 0.054" or maybe 0.053" or 0.055"? Or is it 0.054" but locking the table pushed it to 0.055" Or maybe no table lock is even present on that lathe & its free to drift even more constrained only by the condition of feedscrew + table nut threads which always has some backlash. I cant recall if you set up an independant, accurate gage to monitor this, if so, good thinking. (BTW its not considered good practice to lock the DOC setting & run the threading in reverse to get back to starting position for a variety of reasons).
