Same Data, Different Advice: What AI Can't Settle

In September, a client and I ran the same survey results through two separate AI sessions, theirs and mine, and then compared what we'd found. The survey tested two wordings of the same screen at a consumer search company, and every number matched to the decimal: the average intent scores, the bottom-box scores (the share of people who picked the lowest options), the trust gap between the two versions, even the breakdown of who'd answered. That convergence was the best evidence either of us had that the extraction was right, and getting two analyses to match used to be most of what people argued about.

We still didn't agree on what to do.

They read the result as good news. The two wordings performed about the same, so there was no downside to shipping the new one, and that's true. I read the same numbers as a floor. Intent wasn't just equal across the two versions, it was low in both, with about half the people at the bottom of the scale, and that pointed me at the offer and the price rather than the wording of the screen we'd been testing.

Neither of us made a mistake. The arithmetic was identical, and the whole disagreement lived in what those numbers were evidence of.

Matching numbers only settle the easy half

Two independent passes landing on the same figures tells you the extraction is sound, which is worth knowing. It tells you nothing about what to do, and with AI in the loop it's unusually easy to mistake the first for the second. When two sessions agree to the decimal, it feels like the question has been settled, when only the easy half has.

AI makes that feeling stronger. In October I was at the News Product Alliance Summit in Chicago, where people who build digital products for news organizations spend three days comparing notes, and I sat in on a five-minute talk by Juliana Castro Varon, who leads design for the New York Times' AI initiatives, called "Not a Chatbot." One of her five points was that chatbots want to please you, to a fault.

That tendency is well documented. In research from Anthropic, summarized in AI and Ethics, a model in one experiment backed off answers that had been correct about 98% of the time once a user pushed back. So if I'd taken our disagreement back to my own session and asked which reading was right, I'd mostly have found out which reading I'd typed in with more conviction. I've written before about how AI research tools tell you what you want to hear, and a tie between two people is where that habit does the most damage, because both people are asking.

Annotation researchers treat disagreement as a signal

People who build training data for AI have been working through this for more than a decade. When two careful annotators label the same sentence differently, the old instinct was to treat it as a quality problem: tighten the guidelines, retrain the labelers, take the majority vote. Lora Aroyo and Chris Welty, writing in AI Magazine in 2015, argued the opposite. In their research, "disagreement is not noise but signal," and at the level of a single example it often means the sentence itself is ambiguous or the category is vague. The CrowdTruth researchers later put the cost of the majority vote plainly: it creates "artificial data that is neither general nor reflects the ambiguity inherent in the data."

There's a newer wrinkle now that models do a lot of the labeling. A 2025 study of models used as labelers found they agreed with each other more than they agreed with people, with weighted kappa above .8 in most pairings, and that the places where the models disagreed lined up with the places people disagreed at a correlation of only about 0.1. Those were smaller models than the ones most of us use for client work, so I wouldn't stretch the exact figures. The direction still holds, though: models tend to agree with one another, and when they do split, they don't split where people do.

Two models agreeing is about as cheap as a yes gets, and a cheap yes is how you end up with an expensive no.

A newsroom ran into the same wall

I was at the summit to help run a hackathon, so I missed the panel called "You Had Me at Taxonomy," but attendees kept shared notes and a transcript, and one exchange from the Q&A is my survey problem in newsroom clothes.

Peter Rasmussen, principal AI engineer at The Baltimore Banner, has been building classifiers that label stories by the reader need they serve. Someone in the audience asked the obvious hard question: when editors disagree about the label, whose answer is gold? Rasmussen said the Banner had pulled three or four of its roughly 50 editors to label a test set, and that the primitive solution is majority vote, then see if it's bad.

That's a reasonable place to start when you need one label to train a classifier against, and "then see if it's bad" leaves the door open. But the stories where experienced editors split are the ones telling you something. If two editors who know the audience can't agree on which need a story serves, the category isn't defined yet, and settling that is a decision about the product, not a labeling chore.

Matt Watson, VP of product at Chicago Public Media, answered it the way I would: make sure everyone's working from the same definitions before anyone labels anything. Earlier in the panel he'd said the old newsroom metadata "tells you more about where an editor put the article … less about who it's supposed to be serving." An undefined category in a survey does something similar. It quietly records what the team assumed, and every number downstream inherits that assumption.

Decide what good looks like before the model runs

Mukul Devichand, who edits the New York Times' AI initiatives, gave a Show & Tell talk in the same room as Castro Varon, about reader-facing AI that no editor checks before a reader sees it. The Times already runs some of these, from automated voice to the Wirecutter Finder. His team's method starts with editors defining what "good" means, then evaluating the output against that definition and giving detailed feedback until the quality holds. "It is not enough just to have retrieval augmented generation," he said, and he named a quality marker the team has added alongside relevance and accuracy: faithfulness, meaning whether the output kept the nuance of the source.

Faithfulness is the right word for my survey story, too. Both of our AI sessions were perfectly faithful to the numbers. Only a person could decide whether "flat" meant "safe" or "stuck."

If you do client research with AI, this is your job now

If you advise clients for a living, you already have this skill. When two people on a client's leadership team read the same report and walk out with different plans, you don't average them. You find out what each of them thinks the numbers are evidence of, and that conversation is usually where an engagement earns its fee. AI hasn't changed that. It's made the arithmetic cheap enough that the reading is most of what's left, and the reading is the part your client is paying for.

In practice that means a little more work up front and a little more honesty at the end. Before you run anything, write down what each number would be evidence of if it came back high, flat or low, because a flat result only means something once somebody has decided which question the survey was asking. When you and a client read the same results differently, write both readings down, say which one you'd bet on and why, and name the result that would change your mind. And treat two AI runs agreeing as confidence in the extraction, because that's all it is.

When Fieldway does the extraction behind the scenes for an advisor, that's where the line sits. The numbers are our job, and the reading stays yours, with your name on it.

My client and I matched to the decimal, and whether flat meant safe or stuck was never going to be in the numbers.