Your Agents Don't Need Better Retrieval. Retrieval Is the Problem.

Years ago at Adaptavist I ran the project to consolidate our scattered documentation onto a single platform. The thing that made it urgent was a finding from the flagship product, ScriptRunner: poor core documentation was one of the top reasons people uninstalled the app.

Not missing documentation. The pages existed. Someone had written them, someone had reviewed them, and they sat on a website that anyone could reach. People uninstalled anyway, because at the moment they needed a particular fact, that fact did not arrive.

I've thought about that project a lot in the last month, because a benchmark published in July found the same failure in machines, and measured it precisely enough to make it uncomfortable.

The facts are stored. They just never arrive.

The InMind benchmark, published on arXiv in late July, tests something most agent evaluations skip. It asks whether an agent memory system can apply a stored fact to a question that doesn't resemble it.

The setup is clean. Take a personal fact – a food allergy, say – and bury it in a long stretch of background conversation, the way a real fact enters a real system. Then ask an indirect question whose correct answer changes because of that fact, but which shares no keywords and no obvious semantic overlap with it. A question about a recipe, not a question about allergies.

Six memory systems spanning vector, graph and agentic designs recall those facts on demand at up to 100% – ask directly and they answer. On the indirect questions, none of them exceeds roughly 14 to 16%. Take the same fact and simply place it in the model's context instead of retrieving it, and the backbone model answers 84.0% of those questions correctly.

Same model. Same fact. Same question. The only variable is whether the fact was retrieved or already present, and it swings performance by something like seventy points.

The authors call this the implicit-association blind spot, and their project site puts the benchmark at 125 expert-verified tasks across ten life domains, 113 of them grounded in citable public sources – FDA allergen guidance, OSHA regulations, USCIS travel rules, product recalls. These are not trick questions. They're the ordinary case where knowing something about a person should change the answer you give them.

Why the obvious fix doesn't work

The reflexive response to "our agent doesn't know our business" is to improve retrieval. Better embeddings, better chunking, a reranker, a bigger index.

The researchers tested a version of that directly. Using an embedding with eight times the dimensionality improved target recall for every system – the right documents came back more often – and left the gap essentially intact. Better search retrieved better, and the answers didn't improve.

That result deserves more attention than it's getting, because it rules out the entire category of fix most teams are currently funding. If the failure were retrieval quality, better retrieval would help. It didn't. The failure is structural to the approach: query-conditioned retrieval assumes the memory you need will resemble the query that needs it. Implicit associations violate that assumption by definition, and no amount of tuning fixes an assumption.

What did work was almost embarrassingly simple. A minimal probe that keeps a compact profile visible in context before the query arrives – no search at all – recovered 68.8%. The authors name the open problem plainly: routing, meaning the decision about which facts must stay permanently visible rather than be fetched on demand.

Which is to say the interesting engineering question stopped being how do we find the right thing and became what does this agent need to already know before anyone asks it anything.

Why this extends to your business

InMind tests personal facts in life domains – allergies, travel rules, recalls. It doesn't test enterprise procedure, and the researchers don't claim it does. The extension to your business is mine, not theirs.

It holds anyway, because the failure mode isn't about subject matter. It's about semantic distance between a stored fact and the query that needs it. And organizational knowledge is unusually far apart from the questions it governs. The rule that a certain discount tier requires a second approver lives in a finance policy document. The query that needs it looks like a customer asking whether you can do better on price. Those two things share almost no vocabulary. That's the blind spot exactly, and enterprise procedure is full of it.

A separate July paper argues a related point from the other direction. Kirchdorfer and colleagues propose an architecture for what they call organizational memory, on the grounds that policies, process models and standard operating procedures are encoded in human-oriented artifacts that general-purpose models can't reliably consume. Their proposal splits the work into curation – how knowledge gets ingested, structured, versioned and updated – and consumption, how agents query and apply it. It's an architecture proposal with a proof-of-concept, not an empirical result, and worth reading as a direction rather than a finding.

But the framing is the useful part: your procedures were written for humans, by humans, for reading. Nobody wrote them to be consumed by something that would encounter them one retrieved fragment at a time, out of order, without the surrounding document.

What this means for the work

The industry has spent two years treating knowledge as a search problem. Point the agent at the wiki, at Confluence, at the shared drive, and the reasoning was that if the information is in there somewhere, a sufficiently good retriever will surface it.

The July research says the information being in there is not the hard part, and never was. The hard part is deciding what matters enough to stay in front of the model permanently, and that is not a retrieval decision. It's an editorial one.

Someone has to decide that these fifteen facts about this customer, this account, this regulatory context are load-bearing enough that the agent should never operate without them – and that the other four thousand documents can be fetched when asked for. That's curation. It requires knowing the business well enough to say which knowledge changes answers versus which knowledge merely informs them.

That's not a job you can buy, and it doesn't get cheaper as models improve. It's the same work that made documentation projects hard before any of this existed: someone with judgment deciding what a person needs to know at the moment they need it, rather than making everything available and calling the problem solved.

In practice I'd start by sorting your knowledge into two piles, which is a harder exercise than it sounds. One pile changes the answer – the constraint, the exception, the thing about this customer that makes the standard response wrong. The other pile informs the answer, and can be fetched when the question obviously calls for it. The InMind results suggest the first pile needs to be small, since the recovery came from a compact profile rather than an exhaustive one, and it needs to be permanently present rather than retrievable.

Most organizations I've worked with can't produce the first pile on request. They can point at a wiki. That's the gap, and no vendor is going to close it for you, because filling that pile requires knowing which facts have changed an outcome before – which is knowledge that lives in the heads of people who've been burned, not in any document.

We built a better search engine for the ScriptRunner docs. It helped. What actually moved the number was deciding which things a new user had to encounter whether they went looking or not.

The facts are in your systems. They have been the whole time. What your engineering budget should be buying is the decision about which handful your agent never operates without.