How do I make an AI fact check its own answer
After an AI gives a confident answer, send a follow-up asking it to rate each claim's confidence and cite a source you can verify. The same content returns as a labelled list instead of a smooth paragraph, and any invented statistic gets flagged as low confidence with no verifiable source.
How to do it
- Start from an answer that sounds too certainUse this right after the model replies to a factual question with a confident, polished paragraph. It works best when the answer includes a specific stat or a named study.
- Send the follow-up prompt asking it to grade itselfPaste an instruction telling the model to rate each claim's confidence and cite a source you can verify. This forces it to re-examine its own answer instead of restating it.
- Read the answer back as a labelled listThe reply changes from a smooth paragraph into a list where each claim carries a confidence label and a source. Compare the before and after on the same task.
- Check the low-confidence, unsourced claimsLook for claims flagged as low confidence with no verifiable source — that is where an invented stat or study name gets exposed.
- Save the prompt for reuseKeep the follow-up handy and run it next time an answer sounds too certain, so you can catch confident but unverifiable claims.
Questions that come up
Can the model lie about its own confidence scores?
Yes. A confidence label is the model's self-report, not a measurement, so it can still rate a false claim as high confidence. The real safeguard is the source it cites: open the link or reference yourself and confirm it exists and says what the model claims.
What if it cites a source that turns out to be fake?
That happens, and it is exactly what this check surfaces. Treat any citation as unverified until you open it. If the source cannot be found or does not support the claim, discard the claim even if the model marked it high confidence.
Does this work on every kind of question?
It helps most with factual claims that have verifiable sources — stats, dates, study names, quotes. For opinions, predictions, or open-ended reasoning there is often no source to check, so the confidence labels matter less and your own judgment still does the work.
Full transcript
You asked an AI a factual question, it answered with confidence — but was any of it true? Here's the follow-up that makes the model grade its own answer. Paste: rate each claim's confidence, cite a source you can verify. Same task, two answers. Before: a smooth paragraph with a confident stat and a study name. After: a labelled list — and the invented stat gets flagged: low confidence, no verifiable source. Save this prompt, and run it next time an answer sounds too certain.