There’s a failure mode in agent programs that by no means exhibits up as an error. The agent works nicely for twenty turns, then will get subtly worse — forgets a constraint from earlier, repeats a device name it already made, contradicts one thing it mentioned. No exception, no alert, no apparent trigger.
What occurred is that context stuffed up and one thing needed to go. The query is whether or not you determined what went, or whether or not a default did.
The place the tokens really go
Earlier than optimizing, measure. A protracted-running agent’s context is often 4 issues:
Software definitions. Each device the agent can see prices tokens for its identify, description, and JSON schema. A well-documented MCP device runs 150–400 tokens. Join twelve MCP servers averaging twenty instruments every and also you’re at 240 instruments — name it 30,000 tokens of schema, resident in each single request, earlier than the person has mentioned a phrase.
That is the most typical and most invisible price. It scales with what you linked, not with what the agent makes use of.
Dialog historical past. Grows linearly with turns. Everybody expects this one.
Software outcomes. The variance killer. Most calls return a couple of hundred tokens. Then one question returns 400KB of JSON, and except one thing intervenes, that payload sits in context for the remainder of the run — re-sent on each subsequent flip.
System immediate and directions. Fastened and often the smallest piece, which is ironic given how a lot time groups spend enhancing it.
The distribution issues strategically: two of the 4 are structural (device definitions, device outcomes) and might be fastened as soon as, architecturally. The opposite two are linear and might solely be managed. Repair the structural ones first — they’re the place the massive, low-cost wins are.
4 methods
Deferred device loading
Don’t put all 240 device schemas within the immediate. Give the mannequin a compact index — server names and one-line summaries — and cargo full definitions just for the server it decides to make use of.
Typical saving is 60–80% of tool-definition tokens on runs that contact two or three servers, which is most runs.
The fee: an additional spherical journey when the agent picks a brand new server, and barely worse device discovery. The mannequin can’t cause a few device whose schema it hasn’t seen, so in case your agent genuinely wants to match capabilities throughout many servers, the index needs to be adequate to route on. Write these one-liners fastidiously.
Massive device response offloading
Set a threshold — 5,000 tokens is an affordable default. Underneath it, outcomes go into context usually. Over it, write the payload to the sandbox filesystem and put a abstract plus a path into context as a substitute. The agent can then grep, slice, or parse the file with code if it wants element.
This converts a catastrophic price right into a small one, and it suits how the information is often used. An agent that fetched 10,000 rows nearly by no means wants all 10,000 in its reasoning; it wants an mixture or a handful of matches.
The fee: the agent wants sandbox entry, and it must be prompted to really question the file reasonably than guessing from the abstract. With out that nudge it’ll generally reply from the abstract alone — confidently and wrongly.
Code mode
As a substitute of the mannequin making 5 sequential device calls and receiving 5 outcomes into context, let it write one script that makes all 5 calls, joins the information, and returns solely the ultimate reply.
The intermediate payloads by no means enter context in any respect. For genuinely multi-step knowledge work — cross-referencing two programs, aggregating throughout pages — that is the distinction between a run that matches and one which doesn’t.
The fee: more durable to debug, for the reason that reasoning is inside a script reasonably than seen as discrete steps. It additionally requires the agent to be a reliable sufficient programmer for the duty, which is model-dependent. Use it for data-shaping work, not for selections you’ll have to audit step-by-step.
Sub-agents
Delegate a bounded subtask to a contemporary agent with its personal clear context. It burns thirty turns exploring, and returns one paragraph. The father or mother’s context grows by that paragraph.
That is essentially the most highly effective approach out there and the one folks attain for final. It’s additionally the one which composes with every little thing else: a sub-agent can use deferred loading, offloading, and code mode inside its personal run.
The fee: complete token spend often goes up even because the father or mother’s context stays small, since you’re paying for the sub-agent’s full transcript. You’re buying and selling tokens for high quality and run size. That’s usually the fitting commerce, however watch the invoice, and watch sub-agent depend for delegation loops.
And compaction, as a ground
When context approaches the restrict regardless of all the above, summarize older turns reasonably than dropping them. Truncation discards the choice that explains present state; summarization retains a lossy model of it.
Compaction is a security web, not a method. If it’s firing usually, one of many 4 methods above isn’t doing its job. Deal with each compaction occasion as a sign price investigating reasonably than a function working as meant.
The routing drawback beneath
All of this assumes you possibly can see what instruments exist earlier than deciding what to load — which suggests device definitions want to come back from someplace queryable, not from recordsdata dedicated subsequent to every agent.
That is the place MCP’s design pays off. In case your servers are registered centrally, an agent can enumerate out there functionality cheaply, pull schemas on demand, and choose instruments at runtime. TrueFoundry’s MCP Gateway is one implementation of that registry sample — servers register as soon as, authentication and per-user OAuth keep on the gateway, and brokers choose particular person instruments from {the catalogue} reasonably than embedding endpoints and credentials. The facet profit is that instruments flagged damaging as soon as, centrally, inherit an approval gate in each agent that makes use of them.
The associated level is that these methods are runtime conduct, not utility logic, which suggests they belong within the harness reasonably than in your agent. TrueFoundry’s agent harness documentation enumerates them as configuration toggles — deferred device loading, giant device responses, code mode, subagents, compaction — which is a helpful guidelines no matter what you run on. When you’re implementing these your self, that listing is roughly the scope.
Instrument earlier than you optimize
Monitor two issues per flip: energetic context measurement and context composition — how a lot is device definitions versus historical past versus outcomes.
Dimension tells you ways near the wall you might be and when compaction will fireplace. Composition tells you which of them approach to use. An agent at 80% context that’s principally device definitions wants deferred loading. One which’s principally a single huge device consequence wants offloading. One which’s principally historical past wants sub-agents. Identical symptom, three totally different fixes, and no method to decide on with out the breakdown.
Most groups have neither quantity. Getting the primary one is often a day, and it’ll instantly clarify no less than one bug you’ve been carrying for a month.
Order of operations
- Measure composition. One afternoon. Do that first; it determines every little thing after.
- Deferred device loading. Largest ratio of saving to effort you probably have many MCP servers.
- Massive response offloading. Kills the worst tail case.
- Sub-agents for something with a bounded, delegable subtask.
- Code mode for multi-step knowledge work particularly.
- Compaction as a ground, with alerting when it fires.
Skip straight to step 4 and also you’ll get an actual enchancment and nonetheless be paying 30,000 tokens a request for instruments no person calls.
