Verifying Everything AI Tells You

A language model predicts likely text. A citation is text. So when a model needs a source to finish a sentence, it can produce one with the right shape: a plausible author, a plausible title, a URL on a real domain. Sometimes the source is real. Sometimes it is a confident fiction. The model has no internal flag that tells the two apart, and neither does its tone.

So every link, quote and number that came through an AI tool needs a check against the original before anyone repeats it at a meeting, in testimony or online.

How often it goes wrong

Products improve, and these numbers will age. The pattern has held across studies, though: errors are common, and they arrive in the same calm voice as correct answers.

Real damage has followed. In May 2025, a summer reading list in a syndicated insert printed by the Chicago Sun-Times recommended 15 books. The first ten did not exist, though several were credited to real, famous authors. The freelance writer told 404 Media he had used AI and not checked it. One unchecked list, and two newspapers that ran the insert had to apologize in public.

How it fails

The routine

The standard check is a claim ledger. Every factual claim in a document gets a row. Each row is checked against the original and ends as verified, corrected or cut.

Look at the third row. "Cannot open" and "might not open on time" are different claims. Anyone who heard the meeting would catch it, and so would the director. AI tools smooth spoken language into firmer statements because firmer statements are more common in the text they learned from. This is exactly how a fair account becomes an unfair one.

The fourth row shows the other habit: AI search often points to a news story about a document. The trail ends at the document itself. If the news story got the number wrong, the document shows it.

Link checks, step by step

1. Click the link. Does it load? Is it on the agency's or publisher's own domain?
2. Is it the right document: right year, right agency, final version, not a draft?
3. Search the page (Ctrl+F or Cmd+F) for the number or key phrase. Read the paragraph around it.
4. Does the paragraph support the sentence, including its strength? "May," "about" and "up to" matter.
5. The page or section goes in the ledger.

Quote checks

Any words inside quotation marks must match a recording, a transcript checked against the recording, a document or your own notes from a conversation you had. If an AI tool supplies a quote and it can't be found, treat it as never said. Paraphrasing it doesn't help; a paraphrase of an invented quote is still invented.

Number checks

The check happens on the page: the year, the unit (thousands? millions?) and the column. Any math gets redone with a calculator. Lesson 5 goes deeper on this.

The second pass: Jeff's tools

Two of Jeff's free tools work as a backstop after a hand check. They catch the things tired eyes miss. They do not replace the reading.

• Source Checker takes a URL or pasted text. It fetches every link to see whether it still works, assesses each citation for whether it is the original record or a retelling of someone else's work and whether a better source exists, and lists statements of fact that have no source. It leaves out opinion, argument and firsthand experience from that list. A first check of a heavily cited piece can take a couple of minutes because links are fetched live.
• Article Values Graph maps the ideas in a piece as a graph, with circle size showing how much each idea carries the argument, and colors each idea by how the piece treats it: promising, dangerous, contested, neutral, or an omission a reader would expect. Its grades are backed by quotes from the piece. The page sets it apart from AI detectors: it ignores who wrote the piece and looks at the thinking and evidence. It shows whether a piece leans on an idea the ledger can't support.

A link flagged as a retelling has an original somewhere behind it. A fact listed with no source either has one waiting to be found or should not be repeated. And a big circle in Article Values Graph for an idea with no row in the ledger is an idea carrying weight it hasn't earned.