Ok, with a real keyboard in my hand I'll attempt to be clearer:
Current GPT-like AIs are a statistical model of text. They break down your query into tokens (roughly words, but sometimes word fragments, symbols, etc), then take that set of tokens and return the probability of each token being the next token. Then it rolls the dice to choose a likely word, appends it to the query text and does the same thing again.
The models are characterized by how many individual tokens there are (GPT3 ran about 50k tokens IIRC) and how many tokens the model can accept in a query.
It's spectacular how well the network takes these inputs and generates "meaningful" text. But there is no validation of the semantics - it's statistical retrieval.
Neither in training the model, or in evaluation of the model, is there any validation against the real world. There is no differentiation between truth and parody and outright lies. These continuations are all considered given a weighting just on the appearance of the input text.
The less supporting material there is in the training database, the more likely you are to effectively wind up with a near word-for-word dump of the input corpus.
You got decent answers to machining questions because the input corpus on machining isn't particularly polluted by misinformation, and the sections put together are effectively cribbed together from a small number of sources.
There's value there as a summarizing tool.
But the day someone mixes in some other discipline in which rake angles refer to green cheese cutters, the quality will decrease - call it dataset vandalism, and it's happening today. There is no way, short of careful selection of the training corpus, to avoid this side effect.
Regarding writing code with it, yes, I've done that - it can generate a quick boilerplate, effectively splicing together web tutorials. But that generated code is almost always incorrect, and requires significant work to massage to correctness. Sadly, the users of said code don't always recognize how flawed it is. I liken it to asking an undergraduate for a homework assignment - you'll usually get something that superficially resembles the solution, but you need to check it carefully for correctness around edge cases, even when it appears to work at first glance.
