We Won Awards for Checkout Speed. Customers Waited Longer.

A metric that covers part of the work moves the rest out of sight. A Kmart stopwatch and England's emergency rooms show where to look.

Matthew Stublefield

6 min read

A cashier at a supermarket checkout counter

When I was in high school I worked as a cashier at Kmart, and at some point corporate decided we needed to check people out faster. They were specific about how they'd know. The clock started when the first item was scanned, and it stopped when the cashier marked scanning as done and told the customer the total.

So we took everything out of the customer's basket first. We lined it all up, every item turned and positioned so it was trivial to scan, while the stopwatch sat there not having started yet, and then we scanned the whole thing very fast and closed the transaction.

Our numbers got much better. The customer's actual wait got longer, because they were standing there watching us arrange their things for the benefit of a timer. Corporate was delighted, and we won awards.

Nobody was cheating, which is what makes this story useful. We were doing exactly what we'd been asked to do. We'd been asked to make a particular number smaller, and we were teenagers, and we figured it out.

Goodhart's law only tells you half of it

Most people who've spent time around metrics know Goodhart's law, that a measure stops being a good measure once it becomes a target. It's true, and I've written about it, and it tends to leave people with the impression that the danger is in the numbers getting fake.

The Kmart numbers weren't fake. Every checkout really did take less time, from first scan to total. What happened is that the work moved. Unloading and arranging the basket still had to happen, and it still took time and the customer still stood there for it, but it had moved to a place the measurement couldn't see.

That's the version of this problem that worries me most in software, and it's harder to spot than gaming, because nobody's done anything wrong and the dashboard is telling the truth. When a metric covers only part of the work, the pressure on that metric doesn't improve the work. It relocates the work to the part nobody's counting. So the first question I'd ask about any metric is where its clock starts and where it stops.

The ambulances outside the emergency room

The cleanest large-scale example I know of has nothing to do with software. England set a target that patients in hospital emergency departments should wait no more than four hours, and, as the Institute for Government's review of public-service targets describes it, the wait was originally measured from when patients were admitted into the department.

You can probably see where this is going. A patient who arrived by ambulance and hadn't been admitted yet wasn't on the clock. So, as Gwyn Bevan and Christopher Hood documented, some hospitals had patients wait in queues of ambulances outside until they were confident the person could be seen within four hours. The waiting hadn't gone anywhere. It had moved outside the building, where it tied up ambulances that somebody else might have been waiting for.

Here's why I like this example more than the ones usually told about bad metrics. The four-hour target wasn't a bad target. The same review reports that one analysis found it cut waiting times by an average of 21 minutes, and that the reduced wait came with a 14% reduction in the number of people dying within 30 days of being admitted to the emergency department. It's hard to think of a better reason to measure something.

So nobody threw it out. They moved the start of the clock. The target was amended so that it begins either 15 minutes after the ambulance arrives or when the patient is transferred to the department, whichever happens first, and the ambulance stopped being a free place to put the wait.

That's the fix I wish more product teams reached for. When a metric gets distorted the instinct is usually to get cynical about metrics, or to add four more so nobody can game them, and what you often need instead is to go find the unmeasured place the work has moved to and pull the boundary out far enough to include it.

Where this shows up on a product team

Cycle time is the obvious one. If your clock starts when an engineer moves a ticket to "in progress," then all the waiting that happened before that, in the backlog, in somebody's approval queue, in the debate about whether to do it at all, is the basket being unloaded onto the belt. The team can get faster and faster at the measured part while a customer's request takes just as long as it ever did. And if you start the clock when a pull request opens, the work simply shifts to before the pull request.

Support has a version too. Measure time to first response and you'll get very quick first responses, and many of them will be a polite note saying someone will look into it. Measure conversion on the sign-up page and the friction gets moved to step two of onboarding, where it's somebody else's number. Count features shipped, and you'll get features.

None of the people in these examples are doing anything sinister. If you're a founder wondering why every dashboard is green and nothing seems to be moving, I'd start by assuming your team is as good at this as we were at Kmart, and that they're responding sensibly to what you asked for.

Choosing a metric is a way of protecting your team

The other reason I care about this has less to do with accuracy. I think a big part of a leader's job is protecting their team: their time, their focus, and their reputation. And one of the less obvious ways you do that is by making sure the measures of success are ones that honor your team's work and aren't going to get them thrown under the bus when something else doesn't work out.

A badly bounded metric can do that to people. Imagine you've told a team their job is to reduce cycle time, and the clock starts at "in progress," and they've done it, and the CEO still can't figure out why customers are waiting nine months for anything. That team hit its number. It also just became the most convenient explanation in the building, and it's going to have a very hard time defending itself with a chart that says it did what it was asked.

We weren't in any danger at Kmart, of course. We got awards. But the grown-up version of this, where the number gets met and the outcome doesn't, ends differently, and it's rarely the person who picked the metric who pays for it.

Before you put a number on a slide

Write down exactly where the clock starts and exactly where it stops, in plain words, the way corporate did for us. Then ask the people being measured what happens just before the start and just after the stop, since they've almost certainly already worked it out, and they'll usually tell you if you're asking out of curiosity. After that, look for the person who's experiencing the whole thing, end to end, and measure from where they're standing.

It'll be a messier number and a less flattering one, and you probably won't win any awards for it.

Somebody at corporate looked at those checkout times and decided we'd done something wonderful. The customer was still standing there.

–Matthew

More Field Notes

  1. Consulting Operations

    Product Managers Rate Themselves Lowest on the Calls Their Bosses Get Paid to Make

    Product managers rate themselves strongest on daily work and weakest on commercial judgment. Why that gap persists, and what it means for founders.

  2. Consulting Operations

    AI Agents Raised Pull Requests 36%. Shipped Work Didn't Keep Pace.

    A Harvard study of 718 companies found AI agents raised code output far faster than shipped work. Why, and what a founder should measure instead.

  3. Consulting Operations

    I Cancelled the Test Before We Built It. The Math Said It Couldn't Win.

    A test that can't produce a readable result shouldn't be built, and the time to decide what would make you stop is before the results come in.

Want help running a sharper practice?

The reading and synthesis behind your client work, handled – a living deliverable kept current, so more of your time goes where your name is actually on the line.