Context Builder: why coding agents fail on large repos — and what to build instead
LLMs in coding agents don't fail because they're "not smart enough." They fail because the repo is bigger than the window, exploration is noisy, and session memory dies when the chat ends. A Context Builder is infrastructure for that gap.
Everyone is wiring LLMs into coding agents. Non-tech folks are doing the same with docs, Notion, and Drive. The pattern is the same: the corpus is large, the model sees a slice, and the answer is confidently wrong.
On big codebases the failure mode is especially familiar. The agent greps a keyword, opens two files, misses the real call path two packages away, invents an API that doesn't exist, and burns half the context window on exploration noise — before it writes a single useful line.
So the question isn't "can we make a smarter agent?" It's: can we build a system that generates the right context for the LLM before (and during) the task?
A Context Builder is not another chat UI. Given a task + corpus, it produces a budgeted, structured context pack that an agent can load in one shot — Map, Facts, Evidence — instead of rediscovering the repo every session.
What "good context" actually is
Most tools today do a weak version of retrieval: embed everything, pull top-k chunks, hope. That's not enough. Useful context has layers:
The pack itself should also carry a short preamble ("you're in X monorepo; auth lives in …") and budget metadata: token counts, what was dropped, and why. Without that discipline, "more context" just becomes expensive noise.
Not RAG. Not another explorer.
Two comparisons matter.
Vs plain RAG: task-shaped packs (fixing a bug ≠ adding a feature ≠ explaining architecture), structural retrieval over the call/import graph, living team memory, and explicit token budgets — delivered as agent tools, not a separate chat product.
Vs a normal agent that "reads the repo": session agents already grep and open files. That knowledge is ephemeral, costs many tool rounds, and often misses neighbors. A Context Builder is infra those agents call — index ahead of time, assemble a pack at ask time — so the session starts oriented instead of half-blind.
| Dimension | Session agent | Context Builder |
|---|---|---|
| When | During the chat, tool by tool | Index ahead; cheap pack at ask time |
| Knowledge | Dies with the transcript | Persists in a local structural index |
| Output | Answers / edits | A budgeted context pack any agent can load |
| Goal | Solve this message | Orient the agent with less rediscovery |
| Role | Decision-maker + implementer | Infrastructure the implementer consumes |
The agent still reasons, edits, runs tests, and reads extra files the pack flags as uncertain. The pack is input — not a replacement for judgment.
Differentiation requires at least: a persistent structural index, task-shaped packing with drop tiers, stable Facts injected every time, and a measurable win — fewer tool rounds and higher task success vs explore-from-scratch.
The shape of it
Index the repo once into a structural map. At ask time, plan what kinds of context the task needs, retrieve with more than similarity (graph + lexical + conventions), and pack under a hard token budget. Deliver it as something an agent can call — ideally an MCP tool like build_context — so Cursor or Claude starts oriented instead of grepping in the dark.
First wedge: large code repos, local-first, agent-native. Not a hosted knowledge platform. Not another chat window. Infra that turns “the agent should just know the repo” into a pack it can actually load.
That's the idea I'm exploring. If you're hitting the same wall on monorepos, I'd like to hear what breaks first for you.