AI Fact-Checking: A Practical Checklist for Writers
To be honest, in June 2023, Judge Kevin Castel of the U.S. District Court for the Southern District of New York ordered two lawyers to pay a $5,000 fine. The story was that they had cited judicial cases in a court filing, none of which existed; they were the chatbot’s own inventions. Before submitting, one of the lawyers had asked that same tool, “Are these cases real?” and received the answer, “Yes, they are real and can be found in official databases.” This was not the first such case, nor will it be the last. The Mata v. Avianca case is a small picture of a large problem: AI output is fluent, but fluency is a long way from accuracy.
Why do errors seem so believable?
A language model is trained to predict the next token, not to tell the truth. Its optimization criterion is how similar the next sentence is to human texts; for this reason, a confident tone, tidy structure, and seemingly precise citations are part of the generation style, not a sign of credibility. Emily Bender and Alexander Koller argued in the paper “Climbing towards NLU” (2020) that meaning is not fully present in linguistic form and takes shape only through connection with the outside world. A year later, Bender and her colleagues extended this idea to larger models in “Stochastic Parrots”: skillful repetition of language patterns does not by itself create understanding or correctness.
In practice, three factors exacerbate errors. First, training data is outdated or incomplete, and the model does not always correctly state the cutoff date of its knowledge. Second, compressing information into billions of parameters blurs details; it remembers the journal name, but not the page number. Third, the model is biased toward answering and says “I don’t know” less often than it should.
Checklist: Which claims should we verify first?
Reviewing a two-thousand-word text line by line with the same care as checking a research report is not practical. Prioritizing based on the cost of error is a good place to start. Experience shows that these seven categories cause the most damage:
- Numbers and statistics: Percentages, years, growth rates, and quantitative thresholds are most susceptible to soft fabrication.
- Direct quotations: A sentence in quotation marks whose source cannot be found undermines the credibility of the entire text.
- Bibliographic citations: Plausible-sounding titles with the correct author and the wrong journal.
- Proper names: People, companies, and products that appear together in unusual combinations.
- Legal, medical, and financial claims: Errors in these areas have real consequences for the reader.
- Dates and sequences of events: A reversal of cause and effect easily escapes notice.
- Code and technical instructions: An example that has not been run only looks like code.
Risk and Verification Method Table
| Claim type | Warning sign | Verification method |
|---|---|---|
| Paper citation | Fluent title but no DOI | Search the title in Crossref and Google Scholar |
| Statistics | Round number without a year | Consult the original report from the publishing institution |
| Quotation | No date or place | Find the full text of the interview or book |
| Historical event | Exact date but vague source | Cross-check against a reputable news archive |
Seven verification techniques that actually work
- Go back to the primary source. Reposted summaries and roundup pages multiply errors. Look at the original text or official report.
- Ask the model a reverse question. Ask, “If this claim were false, what evidence would refute it?” and “Which part of it do you not know?” These questions usually draw out useful admissions.
- Two tools, two models. Agreement between two models is not proof of correctness, but disagreement between them is a red flag worth investigating.
Any claim that contains a number, name, or date is an assumption, not a fact, until you have seen its evidence with your own eyes.
Documentation: Where Do We Keep the Paper Trail?
Checks that are not recorded start from scratch again in the next review. A simple file next to the draft is enough: a claim column, a source column, a status column (confirmed, rejected, pending), and a review date column. In the text itself, you can also use temporary markers; for example, the label “[needs verification]” until a source is found. This has two benefits: later versions of the text are published cleaner, and if a reader objects, you know where each sentence came from.
The last point is that text-generation tools produce drafts, not documents. Final responsibility for the sentences rests with the author whose name appears under the text. AI can multiply writing speed, but speed cannot take the place of accountability.
Conclusion
Fact-checking is not a stage to be done at the end; it is part of the writing process. By prioritizing high-risk claims, going back to primary sources, and recording the results of checks, you can turn AI output into text that is both fluent and defensible. The difference between a professional writer and a rushed writer lies in those few extra minutes spent looking at the evidence.
