Study // Knowledge card 2.1
What people say is a hypothesis
How to use Jakob Nielsen's first rule of usability — watch what people do — to weigh what users tell you in interviews and surveys.
- Date
- rev.
- Aim
- When can I trust what a user tells me?
- Topic
- Research & Discoveryprinciple · core
- Built with
- Claude + NotebookLMUpdated · 5 min read
Notes
2.1aThe gap, and where it comes fromsrc 01
The say–do gap · khoảng cách nói – làm is the distance between what people say they do and what they do. Three forces open it:
- Memory edits. People rebuild what they did from memory, and memory is fallible — above all for small interface details they never noticed.
- Politeness rounds up. In front of a researcher, people give the answer they think you want, or the one that sounds reasonable.
- Reasons come after. Asked why they did something, people build an explanation after the fact.
The gap is the normal state, even in a session that went well: Surprising: I expected the gap to come from careless or dishonest users. It comes from how memory and conversation work, so the most careful participant has it too.
In my words
Words are a report about behavior, filtered by memory, manners and hindsight — so I read them as a report and look for the behavior itself.
Why
Does it matter for design? Because all three errors push the same way: toward the tidier, more agreeable answer. A decision built on words inherits that lean.
2.1bRanking what people tell yousrc 01
Nielsen's basic rules, read as a ladder from strongest to weakest:
- Watch what people actually do. This is the data.
- Do not believe what people say they do. A report, one step removed.
- Definitely do not believe what people predict they may do. Confused: If a prediction is the weakest rung, why do so many surveys and interview guides end on 'Would you…?'
A preference between designs is worth collecting only after people have used them. Before that, it reflects surface: what looks appealing. Whether the design works shows only in use.
In my words
Distance from a real act is the measure: the further a statement sits from one, the less it predicts.
Link
2.1a — the ladder is the three forces, stacked.
2.1cThree moves that close most of the gapsrc 01
- Observe first. Watch someone do the thing — in person, in a recording, in the logs. Behavior is the data; words are commentary on it.
- Ask about the past, specifically. "When did you last…?", "Show me the one you sent yesterday." Actionable: First pass on the next interview guide I draft, before any session is booked., because nobody knows what they would do, and everyone says yes.
- Treat a stated preference as a hypothesis. Test it with a prototype before it goes in the deck as a finding.
Why
Does "when did you last" work? Because it asks memory for one real event instead of a forecast, and a real event is harder to round up.
Link
A brief names where the problem shows — a brief reports the symptom; what people ask for is a "say" too.
2.1dAt work: four situations
- "Users keep asking for X." A request is a say. Ask what they were trying to do when they asked, and what they do today instead. Actionable: Backlog grooming: one line on each request ticket before anyone estimates it.
- A survey says most people want a feature. That measures predictions, the weakest rung. Use it to choose what to test, then put a prototype in front of a few of them and watch.
- A participant says "I'd definitely use this." Connection: 2.1b — 'I'd use this' is a prediction, the bottom rung. Ask about the last time they had the problem, and look for a step that costs them something — a follow-up booked, a second visit.
- Someone fails the task, then rates it easy. Report both. The performance answers whether the design works; the rating tells you how they felt about it.
Critique
Goal: decide what to build. Element: a tally of requests. Works? Partly: it measures how often people say it. Rank requests by the workarounds you have seen.
Examples
Two cases of the same shape, kept for reference.
2.1eNew Coke, 1985src 02
Close to two hundred thousand taste tests preferred the sweeter new formula. It launched in April; by July the old formula was back on shelves. The tests measured a sip. Buying was a habit and a memory.
Link
2.1b — a preference collected without real use.
2.1fNetflix, 2017src 03
Five stars were replaced by a thumb. People rated documentaries highly and watched sitcoms.
Link
2.1a — the rating said who people wanted to be; the viewing history said what they played.
Meaning
Sketch (from my case study · Houseplus Post-Purchase)
What staff said
- Interviews with 5 internal employees
- Sales, customer service, debt management
- Buyer maps built from their accounts
What buyers did
- Watch buyers pay an installmentnot done
- Watch buyers work through a procedurenot done
On Houseplus, a post-purchase app for home buyers, I interviewed five internal employees in sales, customer service and debt management. None were home buyers. The buyer-facing empathy and journey maps synthesize their accounts, so I label them as stakeholder accounts: a say, at one remove. What the case lacks is the do — watching buyers themselves pay an installment or work through a procedure.
Paraphrase (my own words)
- Treat words as a lead to check and behavior as the evidence.
- Honest people drift too; the gap comes from how recall and conversation work.
- Rank the evidence: seen behavior first, a prediction last.
- Ask about one recent, real time; ask for preferences after use.
Critique (disagree, edge cases, trade-offs)
- Leading questions. "Would you like a faster way to…?" has one answer. Tell-tale sign: your interview notes agree with your proposal on every point.
- Throwing the words away. The gap is no reason to stop listening. Words show how people think about the thing — their vocabulary, their mental model. Keep both columns; label which is which.
Remember
The note in a few cards: the idea, the words, the steps.
Concept
Words are a hypothesis
Treat what people say as a hypothesis and what they do as the data, because words drift from behavior.
Concept
The evidence ladder
Watching beats a past account, then a preference, then a prediction, because each rung adds memory, manners or guessing.
Term
Say–do gap · khoảng cách nói – làm
How far an account of behavior strays from the behavior itself; memory, politeness and hindsight widen it.
Not Dishonesty.
Term
Stated preference
Which design someone says they prefer; meaningful once they have tried it.
Not What they did.
Workflow
Close the gap
- Observe first
- Ask about the past
- Test the stated preference
Retrieval
Cover the notes. Answer each question from memory, then turn the card to check.
Q1
Why do people's words drift from their behavior, even when they are honest?
Tap to answer
Q1Answer
Why do people's words drift from their behavior, even when they are honest?
Recall rebuilds the event, the setting rewards an agreeable answer, and the explanation is assembled afterward. None of it needs a lie.
Tap to turn back
Q2
Rank from strongest to weakest evidence: a prediction, observed behavior, a stated preference, a specific past account.
Tap to answer
Q2Answer
Rank from strongest to weakest evidence: a prediction, observed behavior, a stated preference, a specific past account.
Observed behavior, then the specific past account, the stated preference, the prediction. Each rung sits one more layer away from an act.
Tap to turn back
Q3
A PM forwards a feature request from a large client. What do you do before anyone designs it?
Tap to answer
Q3Answer
A PM forwards a feature request from a large client. What do you do before anyone designs it?
Log it as a say. Look for the job behind it and the workaround in use now — watch, or ask about the last occurrence — then test a prototype with them.
Tap to turn back
Q4
Your interview guide is full of "Would you…?" questions. How do you fix it, and why does the fix work?
Tap to answer
Q4Answer
Your interview guide is full of "Would you…?" questions. How do you fix it, and why does the fix work?
Ask about the past: the last time it happened, shown on screen. One real event resists rounding up; a forecast costs nothing to inflate.
Tap to turn back
Q5
If words are unreliable, when are they still worth collecting, and for what?
Tap to answer
Q5Answer
If words are unreliable, when are they still worth collecting, and for what?
Always, as a window on how people think — the words they reach for, their motivation. Tag them as a say, and ask which design they prefer once they have tried both.
Tap to turn back
Q1 / 5
Open question
A usability test watches real behavior, but people know they are being watched. How far does that still drift from what they do alone?
Sources
- 01First Rule of Usability? Don't Listen to Users (opens in a new tab) — Jakob Nielsen, Nielsen Norman Group, 2001
- 02New Coke: The Most Memorable Marketing Blunder Ever? (opens in a new tab) — The Coca-Cola Company
- 03Netflix is replacing its five-star rating system with thumbs up and down (opens in a new tab) — The Verge, 2017
Related
Next note