Study // Knowledge card 3.1
Measure the letter the feature exists for
How to use Google's HEART framework, from Rodden, Hutchinson and Fu, to choose the one or two metrics a feature exists to move.
- Date
- rev.
- Aim
- How do I measure whether a design change worked?
- Topic
- Strategy & Metricsframework · core
- Built with
- Claude + NotebookLMUpdated · 6 min read
Notes
3.1aFive letters, one question eachsrc 01
The Google HEART · khung HEART framework names five things worth measuring about an experience. Each letter asks one question:
- Happiness. How do people feel about it? Surveys, ratings.
- Engagement. How much do they use it? Frequency, depth.
- Adoption. How many new people start using it, or a new part of it?
- Retention. How many are still there later?
- Task success. Did they get the thing done? Completion, time, errors.
The letters pull against each other. Faster task success means less time in the product, so Engagement can fall when the design gets better. Note too that Confused: If Happiness is measured by asking, it is a 'say'. Why does the framework put its one stated letter first?; the other four are things they do.
Why
Why not measure all five? Because five charts let any result look like success, and one or two force a direction.
3.1bThe ladder under each lettersrc 01
The letters are the easy half. The other half is Surprising: I expected the five letters to be the framework. The ladder under them is, so a team can know all five letters and still count nothing useful. — the goals–signals–metrics · mục tiêu – tín hiệu – chỉ số process:
- Goal. What should the person achieve? Finish checkout.
- Signal. What behavior shows that? Reaching the receipt screen.
- Metric. How do you count it? Completion rate; median time; errors per order.
The ladder works because a raw count moves for many reasons. Sessions rise when people like a change, and also when they are lost in it. The goal gives the number a direction; the signal ties it to one behavior you can see.
In my words
Each rung narrows the one above it, so by the metric only one behavior is left to count.
Link
What people say is a hypothesis — a signal is something people do; a survey score is something they say about it.
3.1cFour moves from a feature to one numbersrc 01
- Finish one sentence first. "This exists so that people can ___." The letter that sentence belongs to is the one to measure, because it names what the feature is for. Actionable: Put the purpose sentence at the top of my next brief, above its success metric.
- Pick one or two letters. Onboarding exists for Adoption and Task success. Checkout exists for Task success. Fewer letters keep attention on the ones the feature is judged by.
- Write the ladder top-down. Goal, then signal, then metric. Starting from a chart you already have only measures what happens to be easy to count.
- Ask whether more is good for the person. A tool people use all day may not want more Engagement at all. Check the direction before you set a target.
Critique
Goal: pick the letter before any data exists. Element: the purpose sentence. Works? Not alone. An ending as loose as "enjoy it" fits every letter, so the goal under it still has to name something a person can finish.
3.1dAt work: four situations
- "What did the redesign actually do?" A stakeholder wants one answer, and the answer starts from the goal the redesign was for. Name that letter, and report its metric first.
- The dashboard has thirty charts and has never changed a decision. Most of those charts have no goal above them. Actionable: Bring this to the next dashboard review, and archive every chart nobody can tie to a goal.
- A brief's success line says "improve the experience." That line is a wish. I ask what the person should be able to finish, and measure from there.
- "NPS went up." A survey score with no behavior under it. Connection: Same test as the say–do gap: a score is a say until a behavior backs it., and check that the behavior moved too.
Examples
One case of the same shape, kept for reference.
3.1eNetflix, 2017src 02
Netflix replaced its five-star ratings with a thumb in 2017. In a test with hundreds of thousands of new users, the thumb logged over 200 percent more ratings, and that settled the change. Todd Yellin, its vice president of product, argued that what people watch says more than the stars they give.
Link
3.1a — stars are a stated feeling; Yellin put more weight on what people watch, the kind of behavior the other letters count.
Meaning
Sketch (from my case study · MerriTrade)
- HHappinessNot recorded
- EEngagementNot recorded
- AAdoption17,000 subscribed users, per the PM
- RRetentionNot recorded
- TTask successSigned up → played through to the end96% completion
Task success chosen · no written ladder recorded
On MerriTrade I designed the trading contests, from registration to the result screens. The figure that matters for them is completion: among users who signed up for a tournament, 96% played through to the end. In HEART terms that is a Task-success signal, where a sign-up count would only be Adoption. It falls short of the method in two ways. Nothing records a goal–signal–metric ladder, and the rate shows that people finished without showing that the design caused it.
Paraphrase (my own words)
- HEART splits an experience into Happiness, Engagement, Adoption, Retention and Task success.
- A count means something only under a goal and a signal, because counts rise for opposite reasons.
- The ladder runs one way: goal › signal › metric.
- Track just the letters the feature serves, and drop the others.
Critique (disagree, edge cases, trade-offs)
- Engagement by default. Time in app rewards the loop that keeps people inside, even when the extra time does nothing for them. Tell-tale sign: the target says "more" with no word on who gains from it. See Fix one part and the symptom moves.
- It counts without explaining. HEART tells you that Task success fell. The reason needs another method, so pair a moved metric with watching a few people do the task.
Remember
The note in a few cards: the idea, the words, the steps.
Concept
HEART · khung HEART
Judge a feature on its own letters, since a gain in one HEART letter often costs another.
Concept
A count needs a goal
Without a goal a rising number is ambiguous, since delight and confusion both raise it.
Term
Goals–signals–metrics · mục tiêu – tín hiệu – chỉ số
The goal, the behavior that shows it, how to count it: checkout, receipt screen, completion rate.
Term
Signal
A behavior you can see that shows the goal was met, the rung between goal and metric.
Not A survey score about it.
Workflow
From feature to metric
- Finish the sentence
- Pick the feature's letters
- Write goal, signal, metric
- Check the direction
Retrieval
Cover the notes. Answer each question from memory, then turn the card to check.
Q1
Why can “sessions are up” not tell you whether a redesign worked, even when the number is accurate?
Tap to answer
Q1Answer
Why can “sessions are up” not tell you whether a redesign worked, even when the number is accurate?
The same total climbs when people enjoy a change and when they get lost in it. Only a goal and a signal say which rise you are seeing.
Tap to turn back
Q2
Order the three rungs under a HEART letter, and say what each one answers.
Tap to answer
Q2Answer
Order the three rungs under a HEART letter, and say what each one answers.
Goal first, the outcome for the person; then signal, the act that shows it; then metric, the count. Top-down, because starting from an existing chart measures whatever was easy.
Tap to turn back
Q3
A PM asks for one success metric for a new saved-searches feature. What do you do before choosing a number?
Tap to answer
Q3Answer
A PM asks for one success metric for a new saved-searches feature. What do you do before choosing a number?
Name what the feature is for in one sentence, take the one or two letters it points to, and build each ladder from the goal down.
Tap to turn back
Q4
Your redesign review shows five HEART charts, three of them trending up. How do you fix the report, and why does the fix work?
Tap to answer
Q4Answer
Your redesign review shows five HEART charts, three of them trending up. How do you fix the report, and why does the fix work?
Cut it to the letters the feature is for, each with a goal and a signal above its metric. A short report has to show a direction; a full one can always find a green arrow.
Tap to turn back
Q5
When is more Engagement the wrong goal?
Tap to answer
Q5Answer
When is more Engagement the wrong goal?
When more time in the product costs the person something: a tool they must use, or a task that should end fast. Better design can lower Engagement while Task success rises.
Tap to turn back
Q1 / 5
Open question
HEART ties four letters to behavior, but Happiness is still measured mostly by asking. Can a behavior stand in for how people feel, or does it measure something else?
Sources
- 01Measuring the User Experience on a Large Scale: User-Centered Metrics for Web Applications (opens in a new tab) — Kerry Rodden, Hilary Hutchinson, Xin Fu — CHI 2010, 2010
- 02Netflix is replacing its five-star rating system with thumbs up and down (opens in a new tab) — The Verge, 2017
Related
Next note