Our LLM search feature fabricated results — the guardrail said "never fabricate IDs"

8 July 2026

There’s a category of production bug that didn’t exist five years ago. Nothing is thrown. No log line is written. Every dashboard is green. And yet your application is confidently telling users things that are not true.

This is the story of one of those bugs, from a real production system: an LLM-powered search feature that invented search results — with plausible names, realistic values, correct formatting — and showed them to paying customers. It includes the trace, the one-sentence root cause, the design debate where the obvious fix turned out to be wrong, and the three-layer fix that shipped. There’s a runnable reproduction on GitHub if you want to poke at it yourself.

If you’ve shipped anything with an LLM in it — or you’re about to — this failure mode is probably in your codebase right now.

The setup: a normal RAG feature

The product is a construction-industry CRM, a decade old, with hundreds of thousands of project records. We added a conversational search feature: the user types “show me the biggest active hotel projects in Dubai,” we run a real OpenSearch query, render the top five hits as clickable cards, and have an LLM write a short natural-language summary underneath.

Standard retrieval-augmented setup. The retrieval side was rock solid — the five cards were always correct, always real, always clickable. The prose was the problem.

The bug report was almost apologetic, the way reports of weird bugs are:

For some list queries, the AI summary describes ten results. The first five match the cards. The other five… don’t seem to exist anywhere in the system.

Read that again, because it’s worse than a normal bug. The summary wasn’t garbled or vague. It listed entries six through ten with names, values, and locations — all plausible, all formatted exactly like the real ones, all completely invented. A user comparing the summary against the cards saw five verifiable results and five phantoms. A user who didn’t compare walked away believing our database contained projects it didn’t.

The trace: what did the model actually receive?

Debugging an LLM feature starts the same way as debugging anything else: trace the data flow and find out what the failing component was given.

The flow looked like this:

1. The controller runs the search and builds a preview from a paginated provider hard-capped at 5 hits.
2. The summary service caps it to 5 again (array_slice($preview, 0, 5) — belt and suspenders) and builds a tool result for the model.
3. The tool result contains total_results — the true total, say 47 — and a preview array of exactly 5 real projects. Aggregations over the full result set were skipped entirely for list-type queries, so that section was empty.

So the model’s entire knowledge of the result set was: “there are 47 matches; here are 5 of them.”

Now put yourself in the model’s position when the user has asked for “the top 10.” You know 47 results exist. You’ve been asked to describe ten. You’re holding five. Your instructions say to provide “a natural language summary.”

The model did the statistically natural thing: it listed the five real projects, and then continued the pattern. Plausible names, plausible values, plausible locations. Entries six through ten were pure interpolation. The model wasn’t malfunctioning — it was interpolating, which is its job.

The root cause: “never fabricate IDs”

Here’s the detail that makes this story worth writing up. The prompt author had thought about hallucination. There was a rule in the system prompt, and I want you to read it carefully:

“Only use IDs from the reference data below. Never fabricate IDs.”

The model never fabricated an ID.

It fabricated names. And values. And locations. Names are not IDs. The rule had been written for reference-data lookups — country IDs, category IDs — and the model honored it with absolute precision while inventing everything around it.

This is the single most important thing I know about prompt engineering:

An LLM guardrail covers exactly what it says, and nothing more.

A human junior developer reading “never fabricate IDs” generalizes it — “oh, don’t make stuff up.” A model does not generalize your intent. A prompt is a contract, and the counterparty is a hostile literalist with infinite patience.

The literalism also explains why the bug was intermittent — the property that had made it feel spooky. It needed two conditions: the true total had to exceed five, and the user’s phrasing had to invite enumeration (“top 10,” “list the biggest…”). Small result sets never triggered it. Most phrasings never triggered it. The bug lived entirely in the gap between the data the model held and the shape of the question — and nobody tests the gap.

The fix everyone suggests first (and why it’s wrong)

The obvious fix came up immediately: just give the model more results. Fetch 10, show 5 cards, let the summary use all 10. More real data, less need to invent any.

It doesn’t close the gap — it relocates it.

With that change, the summary now names five real projects that aren’t in the clickable list. Think about what the user sees: project names in prose, with no cards to verify them against. From the user’s chair, a real-but-unclickable project and a hallucinated project are indistinguishable — both are claims they can’t check. You get the exact same bug report — “the summary mentions results that aren’t in the list” — except now it’s true data generating the confusion. You’ve laundered your hallucination into a UX bug.

And it doesn’t even stop the fabrication. When the total is 47 and someone asks for the top 20, the model sees 47 exist, holds 10 names, and invents 11–20. The gap is structural; widening the window just moves the threshold where it reappears.

Working through that argument produced the principle that became a design invariant for the whole feature:

The AI may only name what the user can see and verify. named ⊆ visible — enforced by the prompt and by the data you send.

The bug was never “the model has too few results.” The bug was “the model is allowed to make claims the UI can’t back up.”

The fix that shipped: three layers

Layer 1 — rewrite the guardrail like a contract. The prompt now states explicitly: the preview array is the only project data the model has; it must never name, rank, value, or describe any project outside it; the UI already shows those five as cards, so don’t re-list them — synthesize; and if the user asked for more than five, state the true total and say the top matches are shown above.

One subtle diff deserves a callout. The old prompt described the preview as “up to 5 sample projects.” Sample — as in, a taste of the more that exists somewhere, an open invitation to fill in the rest. The new wording: “the 5 highest-ranked matches — the complete set of project-level data available to you.” One word was load-bearing.

Layer 2 — make fabrication self-incriminating. Every real project has a human-readable reference ID, which was already rendered as a badge on the UI cards — but was being dropped from the data sent to the model. We added it, and instructed the model to cite it in parentheses whenever it names a project. Now the model can only cite IDs that exist in its input, so any invented entry would be missing its badge — instantly visible in review, in screenshots, in user reports. Cost: one field. Silent fabrication becomes a loud formatting anomaly.

Layer 3 — feed it facts that can’t be hallucinated. This is the real lever, and the one most teams miss. Ask why the model padded the list: because we gave it almost nothing to say. Five names and a count — thin input begets confabulated output. The fix isn’t more names, it’s server-computed aggregates over the full result set: total combined value, min/max range, status distribution, top areas.

Now the summary can say something genuinely useful — “47 matching projects worth about $12B combined, mostly under-construction hospitality work concentrated in two districts; the five largest are shown above” — and every number in that sentence traces to data we computed and handed over. The model’s job shifted from enumeration (which it will always pad) to synthesis (which is the thing it’s actually brilliant at).

The twist: the twin bug

Before closing the ticket, one paranoid question: does anything else use this prompt?

The same product also has company search — “who are the biggest contractors in Dubai?” It turned out to run through the same controller, the same summary service, and the same system prompt. Same five-item cap. Same missing guardrail. Same hallucination exposure — for company names this time, which is arguably worse: inventing a contractor and attributing projects to it is the kind of output that ends up screenshotted in an angry customer email.

Nobody had reported it yet. We found it because we went looking — and because the exposure was shared, the one guardrail fixed both features at once.

The rule this produces: when you find a prompt bug, immediately audit every feature sharing that prompt. Code bugs have a blast radius you can trace through a call graph. A shared system prompt is shared behavior — its failure modes replicate silently into every feature built on top of it.

Three things to take home

1. Guardrails cover exactly what they say. “Never fabricate IDs” does not cover names. Write prompts for a hostile literalist, and enumerate entities and claims — not just the tokens you happened to think of.
2. The AI may only name what the user can see. Enforce named ⊆ visible in the prompt and in the data you send, and require citable tokens (IDs, references) so fabrication becomes self-evident.
3. A prompt bug is never in one place. Audit the siblings before you close the ticket.

And the meta-lesson: LLM features fail silently, intermittently, and in the gap between what the data supports and what the question implies. Your monitoring can’t see it, your test suite doesn’t cover it, and your users may believe it. Debugging the AI is a production discipline now — budget for it like you budget for the database.

Want to reproduce the bug yourself? The demo repo has a keyless mock mode and a live mode against a real model, with the buggy version on main and the fix on the fixed branch.

This story is a chapter from Change Order — a book about AI-assisted development on real legacy codebases, not todo apps. There’s a video version of this story on YouTube, and more war stories coming: next up, the production email that vanished without a single log line.