The thing that nearly got me was never an answer that looked wrong. It was an answer that came with a source attached.
The failure that actually happens
Ask for something factual and you tend to get a clean answer with a reference. A publication, a report, a section of a regulation. It reads like the work has been done. The reference is doing something specific to you psychologically, which is telling you that somebody already checked.
Then you open it and the source is real, the page exists, and it does not say the thing.
That is the one I think is genuinely dangerous, and it is not the same as making something up. An invented source is almost a kindness, because you go looking, find nothing, and immediately know where you stand. A real source wrapped around a claim it does not support is worse, because the mechanism you would normally use to catch the error is the thing that stopped you looking.
I have caught this in my own work more than once. Not often enough to stop using the tools. Often enough that I now open things.
Why I do not expect this to go away
Everybody's instinct is that the next version fixes it. Mine was too for a while.
I have stopped believing that, for a reason that has nothing to do with the technology improving. These systems are rewarded for producing an answer. Saying nothing scores worse than guessing and being wrong, in almost every setting they are measured in and in almost every setting you will use them in. You do not go back to a tool that keeps telling you it is not sure. You go back to the one that answers.
So the confident wrong answer is not a rough edge being sanded down. It is what the thing is optimized to produce, by us, continuously, every time we prefer the assistant that sounds certain.
I would like to be wrong about this. I am not expecting to be.
Fluency is the whole problem
A tool that failed obviously would be easy to work with. You would see the seams and check.
These do not have seams. The wrong answer arrives in the same register as the right one, same length, same confidence, same clean structure. Nothing in the output tells you which one you are holding. Which means the instinct that normally warns you that something needs a second look has nothing to work with. Your judgment is running on a signal that is not there.
I find this is the part people underestimate. They imagine they will notice. I have been doing this a long time and I do not reliably notice. I have to check as a process, not as a reaction, because the reaction never fires.
Where I put the checking
I sort by what happens if it is wrong, not by how much effort went into producing it. Effort is a terrible guide here, since the most confident and most polished output is often the most fabricated.
Checked line by line, every time, no exceptions. Numbers. Names. Dates. Prices. Anything legal or regulatory. Anything with a citation. Anything going to a client, a customer, or onto a form somebody files. Anything I will be quoted on. These are exactly the outputs that look most finished, which is the trap.
Checked once, at the shape. Outlines. First drafts I am going to rewrite anyway. Summaries of documents I provided and can reread. Reformatting. The risk here is emphasis and tone, not fact, and I can see those.
Barely checked. Brainstorming. Rephrasing something I already wrote. Generating twenty options I am going to judge myself. Anything only I will read.
The asymmetry underneath all of this is simple. The tool pays nothing for being wrong and I pay everything. A client does not experience a hallucinated regulation as a tool failure. They experience it as me being unreliable, and they are right to.
Three habits that do the work
Open the source. Not glance at the citation. Open it and find the sentence. This is the single highest return thing on this list and almost nobody does it, because a reference feels like proof and opening it feels like redundant work. It is not redundant. It is the work.
Ask again somewhere clean. If something matters and I cannot easily verify it, I ask again in a fresh session rather than pushing back in the same one, where it will mostly just agree with me. Two separate answers that contradict each other are useful. Two that match are weaker evidence than they feel like, but a contradiction is a reliable alarm and it costs thirty seconds.
Treat every specific as a pointer. If it hands me a statute, a figure, a date, a name, I read that as go and look this up, never as here is the answer. These tools are far better at telling me what kind of thing to look for than at telling me the thing.
What they are actually good at
I would be misrepresenting my own position if I stopped at the failures, because my daily use is not grudging.
They are reliable when the truth is sitting in front of them. Restructuring something I wrote. Converting a format. Drafting from material I supplied. Finding the contradiction between page two and page nine of a document. Explaining a concept that is written about everywhere. Giving me twenty angles so I can throw out nineteen.
The pattern I have settled on is that they are strong where the answer is in the input and weak where the answer has to be remembered. Rare specifics with one correct value are the worst case, and that is precisely the category people most want to delegate, because looking things up is boring.
Used with that grain, this is the biggest increase in what one person can get done that I have seen in my working life. Used against it, you will eventually put your name on something false and find out in front of a client.
What I would tell someone starting out
Use them. Use them more than you currently do.
And decide now, while nothing is on the line, which of your outputs are in the first category. Write the list down if you have to. Because the moment you need that judgment, you will be in a hurry, the draft will look excellent, and the only thing standing between you and a confident falsehood with your name on it will be a habit you either built or did not.
The time these tools save is real. It is just never the checking.