Study // Knowledge card 7.1
Fluent AI output is a hypothesis until checked
How to use Rafael Hernandez's four quality checks to treat AI output as a hypothesis until it can be traced and checked.
- Date
- rev.
- Aim
- When can I let an AI output into a decision?
- Topic
- AI in Designchecklist · core
- Built with
- Claude + NotebookLMUpdated · 6 min read
Notes
7.1aFluent fabrication, and why it looks rightsrc 01
The typical failure of a language model is fluent fabrication · bịa đặt trôi chảy: output with the shape of a fact and nothing behind it. Three things make it hard to see:
- It writes what fits the prompt. Nothing in that step checks the text, so a citation comes out citation-shaped either way.
- Fluency sounds like confidence. A guess and a fact arrive in the same tone, so reading alone cannot tell them apart.
- It fills gaps with the typical case. It knows only what you gave it; the rest comes back as the usual answer.
The dangerous outputs are the ones that look right, because Surprising: I expected a model's mistakes to read like mistakes. They read like citations, so checking by feel passes the worst ones.
In my words
AI output is a draft in a confident voice, so I read it as claims to check.
Why
Why do the checks start from my inputs? Because a fabricated text carries no sign of its own; only what it should rest on can expose it.
7.1bWhy a fixed list, in a fixed ordersrc 02
A checklist · danh sách kiểm tra fits here because judgment by feel fails exactly where the risk is. A list runs whether an output feels suspicious or not.
The order follows what each check needs:
- Traceability needs only your inputs. It comes first, and a claim that fails it is Confused: Who keeps the hypothesis label on once the line is pasted into a deck?
- Context and interpretation need you. Only you know what the prompt left out, and which lines are someone's conclusion.
- Bias needs people outside the inputs. It asks about what is absent.
In my words
Each check asks for something the text cannot give, so the list catches what reading misses.
Link
What people say is a hypothesis — check 3 is the say–do split again: "users want…" is a report about behavior, and it needs the observation behind it.
7.1cFour checks, in ordersrc 02
Ask them every time, before the output touches a decision:
- Traceability — where did this come from? Ask for the input each claim rests on. Anything that cannot be traced is Actionable: Use it on the next research deck I build from an AI synthesis, before the findings slide goes out.
- Context completeness — what is missing because it was not in the prompt? Add the constraints, the real conditions people use the thing in, the edge cases.
- Data versus interpretation — which lines are observation, which are opinion? A quote is data. "Users want…" is a conclusion someone drew, and that someone may be the model. "My eyes hurt at night" is data; "wants dark mode" is one reading of it.
- Bias and blind spots — who or what does this exclude? Ask for counterexamples and for the people the inputs did not cover.
Why
Why label a claim instead of deleting it? Because an untraceable claim can still be true, and a label leaves it open for someone to test.
7.1dAt work: four situations
- AI summarized your interviews, tickets or competitors. Ask it to trace Actionable: Run it on the last AI summary I relied on, before I cite it again.
- AI drafted a persona, a problem statement or copy. It is the model's default answer wherever your prompt was silent. Run checks 2 and 3: add what was missing, and mark what came from data.
- A stakeholder forwards an AI answer as a fact. It is a hypothesis in a confident voice. Before answering on substance, Connection: Same test as check 1 in 7.1c: a claim is only as good as the input behind it.
- The output holds confidential data. Pasting it into an unapproved tool to check it is a leak. Use only the tools your organization has cleared.
Critique
Goal: know how much of an AI summary I can use. Element: counting how many of its main claims trace to my inputs. Works? Only in part, because a claim can point to a real quote and still misread it. Pair it with check 3.
Examples
One case of the same shape, kept for reference.
7.1eMata v. Avianca, 2023src 01
A New York lawyer filed a brief citing six court cases, with names, dates, judges and quotations. None of the six existed: ChatGPT had written them, and the judge fined the two lawyers and their firm $5,000.
Link
7.1a — fluent fabrication: the error had the full shape of a fact.
Meaning
Sketch (from my case study · uPALMS)
- 01 · Traceability · metFixed input: validated workflows, approval rules
- 02 · Context completeness · met10+ "source of truth" files, one per module
- 03 · Data vs interpretationNot documented
- 04 · Bias and blind spotsNot documented
On uPALMS, an enterprise asset management system, Claude Code generated the UI on my direction. Its fixed input was validated workflows and approval rules, and I had structured the repository as 10+ "source of truth" files, one per functional module. So a generated screen could be traced to a written rule and checked against its module, which I read as checks 1 and 2. The case stops there. The output was UI with no research text in it, and nothing documents a data-versus-interpretation or bias check.
Paraphrase (my own words)
- However smooth it reads, a model's answer stays a guess until checked.
- Nothing in generation tests truth, so invented claims look like real ones.
- Ask four questions in order: source, missing context, data or opinion, who is left out.
- Label what cannot be traced; a person decides what to use.
Critique (disagree, edge cases, trade-offs)
- Letting the model grade itself. "Is this correct?" gets "Yes," because the model has no way to know and every reason to agree. Tell-tale sign: the only check in the thread is another prompt.
- Nothing to trace to. The checks lean on your inputs. An answer drawn from the model's training has no file of yours to trace to, so it stays a hypothesis until a primary source confirms it.
Remember
The note in a few cards: the idea, the words, the steps.
Concept
Output is a hypothesis
Let AI output into a decision only once checked, because smooth prose is no evidence it is true.
Concept
Check by list
Run the same list on every output, because feel only flags what already seems off.
Term
Fluent fabrication · bịa đặt trôi chảy
A fact-shaped claim with no source behind it, such as a citation to a paper that does not exist.
Not Obvious nonsense.
Term
Traceability
Every claim links to an input you can show; a claim without one keeps the hypothesis label.
Term
Data versus interpretation
Data is what people said or did; "users want…" is a reading of it, perhaps the model's.
Workflow
Four checks, in order
- Trace each claim
- Add missing context
- Split data from opinion
- Ask who is excluded
Retrieval
Cover the notes. Answer each question from memory, then turn the card to check.
Q1
Why can an AI output be fabricated and still read as correct, even to a careful reader?
Tap to answer
Q1Answer
Why can an AI output be fabricated and still read as correct, even to a careful reader?
Generation picks likely words for the prompt with no truth test, so a guess comes out as smooth as a fact. Only comparing it against the inputs exposes it.
Tap to turn back
Q2
Name the four checks in order, with the question each one asks.
Tap to answer
Q2Answer
Name the four checks in order, with the question each one asks.
Traceability (what is the source?), context completeness (what did the prompt leave out?), data versus interpretation (observed or concluded?), bias and blind spots (who is missing?).
Tap to turn back
Q3
In a planning meeting, a product manager quotes a chatbot's claim about user needs as settled. What do you do before debating it?
Tap to answer
Q3Answer
In a planning meeting, a product manager quotes a chatbot's claim about user needs as settled. What do you do before debating it?
Treat it as a hypothesis and run check 1: ask what it rests on. If it traces to nothing anyone can check, say so, and get it labeled before anyone builds on it.
Tap to turn back
Q4
An AI persona in your deck mixes quotes with "users want…" lines. How do you fix it, and why does the fix work?
Tap to answer
Q4Answer
An AI persona in your deck mixes quotes with "users want…" lines. How do you fix it, and why does the fix work?
Split it: mark each line as observation or interpretation, trace the observations to their inputs, and label the rest a hypothesis. Readers can then see which lines are evidence and which are a conclusion — possibly the model's.
Tap to turn back
Q5
When do the four checks stop protecting you?
Tap to answer
Q5Answer
When do the four checks stop protecting you?
When the claim comes from the model's training, with no input of yours behind it, or when the model checks its own answer. Then only a primary source outside the model can settle it.
Tap to turn back
Q1 / 5
Open question
The checks assume a person reads every claim, but AI output keeps getting longer. How far can a human check scale before it turns into skimming?
Sources
- 01Mata v. Avianca, Inc. — Opinion and Order on Sanctions (opens in a new tab) — U.S. District Court, Southern District of New York (Castel, J.), 2023
- 02Rafael Hernandez's AI-assisted product design process — Rafael Hernandez, Lead Product Designer
Related
Next note