I Cancelled the Test Before We Built It. The Math Said It Couldn't Win.

A test that can't produce a readable result shouldn't be built, and the time to decide what would make you stop is before the results come in.

Matthew Stublefield

8 min read

A braille typewriter on a desk

We had the design, the ticket and the mockups ready to go. A client wanted more of its web audience inside its mobile app, where conversion ran several times higher than on the web, and the next move looked obvious: a better prompt asking people to install the app. Nobody on the team doubted the logic, and I didn't either.

Before anyone built it, I ran the arithmetic on the prize instead of the mechanism. Say the new prompt did ten times better than the old one, which would be an absurdly good result and far beyond anything the previous test had suggested. That works out to a couple dozen subscriptions a month, against a baseline the company already had many times over. Inside the nine-day window we had for the experiment, it came to fewer than two conversions per test cell, and no test can tell two conversions from noise.

So the experiment couldn't succeed, because even if the prompt worked spectacularly, the result would have been unreadable. The work went in the bin, and the effort moved to a surface that reached roughly thirty times the audience, where the same question could actually be settled.

It was about as cheap as a stop gets, since nothing had been built, nobody had a stake in it yet, and the decision took an afternoon of arithmetic. Most stops come later and cost more, and they bruise more people, because by then the bet has a team and a sponsor and a few months of momentum behind it.

Size the prize before you design the test

Conversion-rate thinking hides something simple: a big multiple on a small number is still a small number. A 10x lift sounds like a home run in a planning meeting, and a percentage feels like progress in a way an absolute count never does. So before I design a test, I work out the best realistic outcome in absolute numbers and ask whether that number could clear the noise in the time we actually have. If it can't, the experiment can't tell us anything, and we've learned what we needed without building a thing.

This is the first stop signal, and it's the one most teams skip. It's also the cheapest research there is. Every week or two I research a business idea of my own, and nearly every time I learn that it's a terrible, unprofitable idea. The last one cost me $7.22 to rule out, which beats months of building and testing.

Why teams keep funding bets that aren't working

The harder stops come after the work has started, and there's a well-documented reason they're hard. In 1976, Barry Staw gave 240 business students a simulated investment decision and then showed them how it turned out. The students who had personally chosen the losing option committed the most additional money to it, more than students who'd inherited the same bad decision from someone else. He called it escalating commitment. The person who championed a bet is the person least able to call it off, because stopping means admitting the first call was wrong, and nobody enjoys doing that in front of the people who backed them.

That's why a stop signal has to be agreed before the results come in. Once the data arrives, everyone reads it through what they need it to say. A condition written down in advance turns "should we stop?" from a referendum on someone's judgment into a check against a number.

Linden Innovation describes how Cisco handles this: teams try to shoot down their own idea before it ever enters a funding round, and they define what failure looks like before the project begins, while nobody is attached to it yet. Every week a bet runs without that definition is a cheap yes, and cheap yeses add up to an expensive no.

What a stop signal looks like

A stop signal is a specific result, by a specific date, that everyone agreed in advance would make you stop or change course. It's written down where the people funding the bet can see it. And someone other than the bet's champion gets to read the result. Dennis Kennedy puts that last part bluntly: "If the person holding authority to stop the project is also responsible for promoting it, the option to abandon will almost never be exercised."

When I run this with a team, I gather enough to form a real hypothesis (not certainty, just a hypothesis) and take it to the team: here's what I've observed, here's what I think is actually going on, and here's what I suggest we try. Then we run a short experiment with the signals defined in advance, so we'll know fairly quickly whether the hypothesis was right. If the results don't show up, the hypothesis was wrong, and I'd rather learn that in three weeks than three quarters.

"People keep asking for more" isn't a stop signal

I spent three days in Chicago last month at the News Product Alliance Summit, and one of the panels I missed (I was in another session) was about exactly this: how newsroom leaders decide what to invest in, when to scale and when to stop. The shared transcript is worth reading for how candid the panelists were.

Biswajit Ganguly, CTO of The Baltimore Banner, talked through News Detector, a tool that scans local news sources and flags potential stories for editors. The beta was supposed to take two months and took six. The first version had no interface at all; it was an Excel sheet that got sent around for about four months while the team collected feedback, and its accuracy climbed from roughly 70% to, in his words, "maybe 95 at this point." Asked whether he'd ever considered stopping, he said, "I'm supposed to say no. The truth being told is yes."

What kept it alive was that the newsroom kept asking for new things, which he read as proof the existing version was already worth something. That's a reasonable continue signal, and he'd also tied each AI tool to the KPI of the department using it. What I didn't find in the transcript was the other half: the result that would have made him stop. Requests for more are what enthusiasm looks like, and enthusiasm shows up for things that are new as reliably as for things that are valuable. In June, Ganguly was still waiting on feedback about which of the tool's leads had turned into published stories.

Yukari Kane of the Prison Journalism Project told a shorter story with a clearer stop. A director role drew nearly 700 applications, about 95% of them AI-generated, and she used AI to screen them. Three rounds of interviews later there was still no hire, so for the next senior search she went to an outside recruiter for a shortlist built by people. Jan Schüßler of ZDF described the version I'd most like teams to copy, because his bar is a standing rule rather than a call made after the results arrive. ZDF judges AI work on the quality of the result, independently of the technology, and "if the end result shows a decline in quality, it's simply not an option."

When the result comes back flat

One session I did make it to in Chicago was about Chicago.com, presented by Sylvia Borowski of Chicago Public Media and Austin Smith, president of Alley, one of the firms that built it. Afterward I asked Austin what to do when a short experiment comes back flat and leadership has to hear about it.

His answer depends on the culture. Where it's okay for experiments to fail, "just tell it like it is." It gets harder where leadership reads a flat result as a bad one, and there, as a consultant, he'd "remind them of the ground rules they agreed to when running the experiment." That's a stop signal's second job. The terms everyone agreed to before the results came in protect the people who ran the experiment, not just the budget.

He added something I'd pass along to anyone managing up. It's easy to come up with a laundry list of reasons something didn't work, like not enough staff or the wrong design, and it's better to "frame those reasons as opportunities to sort out so we can try a new experiment." He sees that as "a good way to navigate from a culture of recrimination to one of cooperative iteration."

Conviction bets still need a date

Some bets get made on conviction rather than a revenue target, and I don't think that's wrong. A conviction bet still needs a date and a signal, or it isn't an experiment at all. It's a commitment, and that's fine too, as long as everyone calls it one and funds it like one. I wrote about what that looks like in organizations that do it formally in software teams don't have a gate, and about what narrowing a bet can look like when the answer is "less" rather than "stop."

The stop itself matters too, because how you end a project decides whether anyone volunteers for the next risky one. A stop that was agreed in advance is easy to end well, since nobody has to lose an argument for it to happen.

The cheapest experiment is still the one you cancel before you build it.

Sources

–Matthew

More Field Notes

  1. Consulting Operations

    Competitive Intelligence Just Became the No. 1 Research Priority. Cadence Decides Whether It Lands.

    Demand for competitive intelligence is rising, and three datasets agree on what makes it land: how often it reaches the people deciding.

  2. Consulting Operations

    Before We Built a Resume Builder, We Asked Whether a Resume Was Even the Right Thing to Build

    Before building a resume builder for young job seekers, we asked whether a resume was the right thing to build. The answer changed the product.

  3. Consulting Operations

    Kagi Stopped Building Orion for Windows. Narrowing a Bet Is a Strategy, Not a Retreat.

    Two software companies pulled back from a market and a beloved product in October. How to tell a deliberate narrowing from a quiet stall.

Want help running a sharper practice?

The reading and synthesis behind your client work, handled – a living deliverable kept current, so more of your time goes where your name is actually on the line.