Your knowledge base is too big for AI agents
7 min read
Ask five people on a small team what the product does and you get five different answers.
The designer knows the screens. Sales knows the pitch. Each engineer knows the part they built. The founder knows the roadmap, which is often a list of what probably shipped. None of these match each other, and when someone checks, none of them fully match the code.
Features get built fast, usually by one person and an AI agent. The reasoning behind most of them stays in a chat session nobody saved.
This used to be a small problem. An old doc confused one new hire, they asked someone, and that was the end of it. Now every coding agent, support bot and AI assistant on the team reads the same doc. Agents don't ask. They read the old page and answer as if it were true.
The fix is to write the whole product down in one place, in a form both people and agents can use. Below is the problem, then four steps to fix it.
The problem
Your docs now have two readers: people and agents. People skim and fill the gaps from memory. Agents only know what is written where they can read it.
Yes, every user must add a phone number.
When an old page disagrees with a newer Slack thread, the agent can't tell which one is true. It picks one.
This is common. Unblocked, which sells a tool in this space, talked to 194 engineering teams about how they feed context to their agents:
- About 6 in 10 had built their own setup: search over their docs, a Slack answer bot, rules files, a pile of MCP servers.
- About 3 in 4 told the same story. They built it quickly and got early wins. Then the docs went stale and the person who built it moved on.
- The top problems were stale content with no owner, wasted tokens, and no rule for which source wins when two disagree.
This costs more than it looks. Arpit Bhayani points out that AI coding doesn't make teams ship 10x faster, because of Amdahl's law. If writing code is 30% of the path from idea to production and AI makes that part 10x faster, the whole path only gets about 1.37x faster. The slow part is still deciding what to build and agreeing on it. A shared, true picture of the product helps with exactly that part.
Most teams try one of these, and each solves only half the problem:
| What teams try | What it misses |
|---|---|
| Search over all docs (Glean, Notion AI) | Finds a page. Doesn't make the page true. |
| Experts approve every answer (Guru) | Someone has to review answers every week, forever. |
| Docs in Git, reviewed like code | Great for engineers. Your designer will never open a pull request. |
| Agents keep the docs updated (OpenWiki) | Drifts and gets noisy without a person checking. |
| Small instruction files (CLAUDE.md, AGENTS.md) | 200 lines can't explain how every feature works. |
So there are two problems. The docs are wrong, because nobody wrote down what the code actually does. And once you do write it all down, the docs are too big for an agent to read in one go.
The four steps below deal with both. Steps 1 and 2 make the knowledge base true. Step 3 makes it small enough for an agent. Step 4 keeps it true.
Step 1: Read the code, not the old docs
An old feature list is often weeks out of date. Skip it and go to the code, which is the one source that can't be out of date.
Run several AI agents through the code at the same time, each covering a group of features. Each agent reads the frontend forms and the backend data, then writes one page per feature with the same six sections: what it does, what the user is asked, how it works, every state the screen must handle, limits and edge cases, and where it is in the code.
With seven agents, a mid-sized product takes about an afternoon. One run like this produced 20 feature areas and 353 feature pages.
The fixed sections matter. "Every state the screen must handle" tends to be the most useful one. A designer can open any feature and see every field, error and empty state a customer can hit, without anyone writing it by hand.
Mark anything an agent did not check against the code as unverified. Agents sometimes disagree with each other, and the mark tells you which claim to check first.
Step 2: Write down where your promises and your code disagree
When you read the code instead of the docs, you find the places where the product says one thing and does another.
None of these are lies. Each one was true once, or planned, and then the code changed and the copy didn't.
Put them all on one page called "Promises vs reality." Sales reads it before a call. Design reads it before changing copy. A line comes off when someone fixes the copy or the code.
If you only do one thing from this post, do this. These gaps can sit for weeks, and a customer can hit any of them.
Step 3: Split it into three levels
Now the knowledge base is true, but it is big. Those 353 feature pages come to about 260,000 words. An agent shouldn't load all of that, and no person will read it.
So split it into levels. This is called progressive disclosure: start small and open the detail only when you need it. Anthropic recommends it in their context engineering guide.
| Level | What it holds | Size |
|---|---|---|
| 1 | What the product is, six rules that apply everywhere, one line per feature area | about 650 words |
| 2 | One summary per feature area: the flow, the screens, five must-know rules | about 300 words each |
| 3 | The full feature pages | about 260,000 words |
Level 1 is 0.25% of the whole knowledge base. An agent working on one feature reads level 1, then that area's summary, and opens a feature page only if the task needs it. You can try it here:
Every level links to the next one down, so nothing is lost. It just isn't loaded until it's needed.
Keep level 1 in the code as a file called llms.txt, and point CLAUDE.md and AGENTS.md to it. Copying product rules into every project is worse, because the copies drift apart.
Step 4: One place is the truth, everything else is a copy
The survey's top problem was that there was no rule for which source wins. So pick one.
Choose the tool your whole team already opens every day, so design and sales can edit it too, not only engineers. Everything else, including the Markdown files your agents read, is a copy made from it.
Without this, every team builds its own bot over its own docs, and the bots disagree with each other.
Two more things. A person should read each agent-written page once. And make sure the agents read your latest code, not an old local copy, or they will document screens that no longer exist.
What is still unsolved
- Keeping it fresh is still a people problem. Agents made it cheap to write everything down once. They didn't make anyone own it. One Hacker News commenter put the risk this way: "Everyone is writing. Nobody is reading."
- Fully hands-off updates. Tools like OpenWiki attach evidence to every claim and flag it when the code changes. That looks like the right direction, but someone still has to review the flags.
Try it yourself
- Pick the one place people will actually read. Make it the only place anyone edits.
- Have agents read the code and write one page per feature with the same six sections: what it does, fields, steps, states, limits, code location.
- Start a "Promises vs reality" page and add every mismatch you find.
- Write a level 1 index of under a page. Point every CLAUDE.md, AGENTS.md and bot at it.
- Make updating the page part of shipping. When a feature changes, its page changes that week. When the docs and the code disagree, the code wins.
None of the tools here are new. What changed is that anyone can now point agents at their code and have it all written down in an afternoon. The hard part is keeping it true week after week.