Study // Knowledge card 6.1
A surprise in testing is a Define problem
How to use the Stanford d.school's design thinking stages as a loop, so a surprise in testing sends you back to the problem.
- Date
- rev.
- Aim
- When a test surprises me, which stage do I go back to?
- Topic
- Process & Collaborationmethod · core
- Built with
- Claude + NotebookLMUpdated · 6 min read
Notes
6.1aSix stages, each ending on an outputsrc 01
Nearly every project fits six stages: design thinking · tư duy thiết kế's five, plus the one that gets the work out the door. Each ends on an output:
- Empathize → insights with Confused: How many observations does it take before an anecdote becomes an insight?
- Define → one problem statement · phát biểu vấn đề.
- Ideate → many options, then a few worth building.
- Prototype → the cheapest thing that can be tested.
- Test → what surprised you.
- Deliver → specs, a decision log, and the metric you will watch.
Each stage asks a different question. A debate that belongs to a later stage cannot settle while an earlier one is still open, because nothing yet says what right looks like.
In my words
I read the outputs as handoffs: each one is what the next stage works from, so a missing output stalls the stage after it.
Why
Why add Deliver to the five? Because the decision log and the metric are what the next loop starts from. Work that stops at Test leaves nothing to measure.
6.1bWhy the line is really a loopsrc 01
In practice the stages repeat. One rule closes the loop: a surprise in Test goes back to Define.
- Built from Define. The prototype · bản mẫu was made to solve the problem as the statement wrote it.
- A surprise tests the statement. When people do something you did not expect, the statement's picture of them was wrong or incomplete.
- Polishing aims at the old target. Improving the prototype you are attached to Surprising: I expected a failed test to mean a weak prototype. It can point at the brief instead, so I look upstream before touching any screen.
Why
Does every bad result reopen Define? No, because if people did what the statement predicted and still struggled, the problem stands; that goes back to Prototype. Only behavior the statement did not predict reopens it.
Link
A brief names where the problem shows — Define is that note's whole job: turning the symptom in a brief into one problem statement.
6.1cFour moves that keep the loop honestsrc 01
- Name the stage you are in. One word at the top of the brief: Define, Prototype, Test. Actionable: Try it at the next review that goes in circles, before anyone presents.
- End each stage on its output. A statement, a shortlist, a test plan. An output can be checked before moving on.
- Prototype the cheapest testable thing. Paper, a clickable frame, a script read aloud. Cheap things get thrown away when Test says so.
- Write the surprise down before the test. One line in the test plan names the result that would Actionable: Use it in my next usability test plan, before recruiting starts.
Why
Does writing it first help? Because a result I predicted on paper cannot be explained away afterward, and one I did not predict is easy to spot.
6.1dAt work: four situations
- The team argues about colors before anyone agrees on the problem. It is an Ideate debate running inside Define. Write the problem statement first; color waits until everyone agrees to it.
- "The test failed." A result with no stage is noise. Ask what people did that we did not expect, and let that answer pick the stage.
- "We know the users." You know the last version's users. Tell-tale sign: Connection: Same test as the say–do gap note: what a team remembers about its users counts as a say until someone watches them again. Go back to Empathize before Define.
- Feedback arrives mid-project. Name its stage before answering it. If it is earlier than yours, go there next.
Critique
Goal: keep mid-project feedback from derailing the work. Element: naming the stage it belongs to. Works? Only in part, because it sorts the feedback without saying whether it is right. Pair it with the evidence behind it.
Examples
One case of the same shape, kept for reference.
6.1eIDEO's shopping cart, 1999src 02
A television crew followed IDEO for five days while it redesigned the shopping cart. Day one went to watching shoppers and stock clerks before anyone sketched; the problem came back as a cart you can leave, that holds a child safely, that nobody wants to steal. The day-five cart had a child seat, removable baskets and wheels that turned in place.
Link
6.1a — Empathize ended on evidence and Define on one statement; every feature answers one of its three lines.
Meaning
Sketch (screenshot / diagram)
- Empathize
- Define
- Ideate
- Prototype
- Test
- Deliver
Test → DefineA result that surprises you sends you back to the problem statement first.
Paraphrase (my own words)
- Six stages, each ending on an output I can point at.
- Each stage asks its own question, so naming the stage settles most arguments.
- A surprise in Test means Define got people wrong; polishing the prototype only sharpens the old answer.
- Before each test, I write the result that would send me back.
Critique (disagree, edge cases, trade-offs)
- Stages as gates. A sign-off at each stage makes going back feel like failure, so nobody does, and the surprise in Test gets explained away. Tell-tale sign: the decision log never records a stage reopened.
- Testing to confirm. If the test plan has no way to surprise you, it is a demo. Tell-tale sign: every task follows the path the prototype was built for.
- No rule for when to stop. The loop says when to go back, never when the evidence is enough to ship. Set the Deliver metric before Test, so "enough" is something you can check.
Remember
The note in a few cards: the idea, the words, the steps.
Concept
Surprise goes to Define
Unexpected behavior reopens the problem, because it shows the statement misread people.
Concept
Six stages, six outputs
Each stage ends on one output, because an output is something the team can check.
Term
Design thinking · tư duy thiết kế
The d.school's five stages, Empathize to Test; a sixth, Deliver, ships the work and sets the metric.
Term
Problem statement · phát biểu vấn đề
Define's one output; the prototype answers it, and a surprise shows where it misread people.
Not A symptom from the brief.
Term
Prototype · bản mẫu
The least costly version people can try, such as paper or a clickable frame, cheap enough to throw away.
Workflow
Keep the loop honest
- Name your stage
- End on an output
- Prototype the cheapest thing
- Write the surprise first
Retrieval
Cover the notes. Answer each question from memory, then turn the card to check.
Q1
Why does a surprise in Test point back to Define, even when the prototype has visible flaws?
Tap to answer
Q1Answer
Why does a surprise in Test point back to Define, even when the prototype has visible flaws?
A prototype only answers the problem it was handed. When people act in ways nobody foresaw, the question itself was off, and fixing flaws just sharpens the wrong answer.
Tap to turn back
Q2
Name the six stages in order, with the output each one ends on.
Tap to answer
Q2Answer
Name the six stages in order, with the output each one ends on.
Empathize → evidenced insights; Define → one problem statement; Ideate → a shortlist; Prototype → something cheap to test; Test → the surprises; Deliver → specs, decision log, metric.
Tap to turn back
Q3
Your PM reads the session notes and says, "We need to fix the onboarding screens." What do you ask before anyone changes a screen?
Tap to answer
Q3Answer
Your PM reads the session notes and says, "We need to fix the onboarding screens." What do you ask before anyone changes a screen?
Did people do anything the problem statement failed to foresee? If so, the statement is in question and Define comes first. If they behaved as predicted and still struggled, the prototype is the fix.
Tap to turn back
Q4
Your usability script only walks people down the happy path. How do you fix it, and why does the fix work?
Tap to answer
Q4Answer
Your usability script only walks people down the happy path. How do you fix it, and why does the fix work?
Before the first session, name the result that would reopen Define. Once that is on paper, a surprise cannot be argued away afterward, so the test is able to fail.
Tap to turn back
Q5
When is it right to go forward from Test instead of back?
Tap to answer
Q5Answer
When is it right to go forward from Test instead of back?
When people did what the problem statement predicted and the open issues are about execution. Then the problem holds, and the work moves to Prototype or Deliver against the metric set in advance.
Tap to turn back
Q1 / 5
Open question
The loop says a surprise goes back to Define, but a team with a fixed ship date pays for every step back. How large must a surprise be before reopening the problem is worth that cost?
Sources
- 01Design Thinking Bootleg (opens in a new tab) — Stanford d.school (Hasso Plattner Institute of Design)
- 02The Deep Dive — IDEO redesigns the shopping cart (opens in a new tab) — ABC News Nightline, 1999
- 03Reimagining the Shopping Cart (opens in a new tab) — IDEO
Related
Next note