Playbook, verification

Three Checks For A Confident Wrong Answer

A two-minute routine that catches fluent, sure-sounding mistakes before they reach your work.

The Curio Team · 2 min readWritten with AI from the sources below and checked against them before publishing.

An AI can sound certain and still be wrong. Fluency is not truth. Language models predict likely-sounding text, so a smooth tone tells us nothing about whether the underlying facts hold. We built a three-check habit that catches most confident errors in under two minutes, and it needs no special tools.^{1 2}

The reason confident wrong answers slip past us is simple. We read tone as a proxy for accuracy, and models are trained to sound helpful and sure. So the fix is not to feel our way to the truth. The fix is a short, repeatable procedure we run the same way every time.²

Run these in order. Stop early if a check fails. The goal is not to trust the model less every time, but to catch the specific answers that would cost us if we were wrong.

1. Ask for sources and confidence. Make the model show its work. Request a numbered list of specific sources plus a confidence rating for each claim. Vague citations, dead-looking titles, or a sudden drop in confidence are your first tell.
2. Re-ask the same question a different way. Change the framing, swap the wording, or ask for the opposite. If the two answers disagree on any fact, the model is guessing, not knowing. Consistent answers across rephrasings are weak evidence for, but disagreement is strong evidence against.
3. Spot-check one factual claim yourself. Pick the single load-bearing fact, the one number or name the whole answer rests on, and verify it against a primary source. One real check beats ten more questions to the same model.

Notice the asymmetry we lean on here. Passing all three checks does not prove an answer is right; it only means we found no crack. Failing any one check is close to proof that something is off. That is the useful direction. We are hunting for reasons to distrust, not reasons to relax.¹

One caveat worth keeping. A model's stated confidence is itself a generated number, not a calibrated probability, so treat it as a rough signal and not a guarantee. The same goes for the sources it lists. Always click through; models can invent citations that look perfect.¹

We do not run this on everything. That would be exhausting and pointless. Save the three checks for answers that carry weight: a medical dose, a legal deadline, a number going into a report, a fact we are about to repeat to someone who trusts us. Low-stakes trivia can stay unchecked.

Here is the mindset shift that makes this stick. We treat the model as a fast, well-read colleague who sometimes bluffs. We would never publish a colleague's claim without a glance at the source, and the same courtesy protects us here. Three checks, two minutes, far fewer confident mistakes making it into our work.