Plugin for managing context window size via pruning and summarization.
When the conversation exceeds a configurable token threshold, this plugin first prunes oversized tool results (free, no model call) and only pays for an LLM summarization if the transcript is still over budget afterwards.
Usage
agent = Agent.new("openai:gpt-4",
plugins: [Nous.Plugins.Summarization],
deps: %{
summarization_config: %{
max_context_tokens: 170_000,
keep_recent: 10,
summary_model: "openai:gpt-4o-mini"
}
}
)Configuration (via deps)
:max_context_tokens- Token threshold to trigger compaction (default: 100_000):keep_recent- Number of recent messages to keep intact (default: 10):max_result_chars- Tool results longer than this are truncated before any model call is considered (default: 8192):summary_model- Model string for the summarization call (default: the conversation's own model, which is also the only setting that preserves the provider's prefix cache — see below):summary_count- Read-only counter of completed summarizations:compaction_in_progress- Read-only marker; see "Crash visibility"
Compaction pipeline
- Prune oversized tool results in place. This never changes the message
count or order, so it can never break a
tool_call/tool_resultpair. - Re-estimate the transcript. If pruning alone brought it under the threshold, stop here — no model call, no cost.
- Otherwise split off the recent messages and replace the older ones with a model-written summary.
The pruned messages are kept whichever branch runs, and whether or not the summarization call succeeds: they are strictly cheaper than what they replaced.
Where the logic lives
This plugin is the live compaction entry point — it is what actually runs
during an agent run. Nous.Transcript is the shared library it calls for
the parts that must behave identically everywhere: pruning
(Nous.Transcript.prune_tool_results/2), tool-pair-safe boundaries
(Nous.Transcript.balance_tool_call_boundary/2) and token estimation
(Nous.Transcript.estimate_messages_tokens/1). There is deliberately only
one implementation of each; this module holds no boundary logic of its own.
Prefix-cache friendliness
The summarization request replays the conversation's own system prompt, tools and messages verbatim and appends the instruction as a final user turn. Providers key their KV prefix cache on the literal request prefix, so a bespoke summarizer prompt with no tools would miss the cache entirely and re-bill every token of the conversation being summarized.
Only the response TEXT is kept: reasoning and tool calls are discarded, so
a summary can never plant an unanswered tool_call in the transcript.
Setting :summary_model to a different model gives up that cache reuse,
and — because the conversation is replayed verbatim — sends the raw
conversation, tool results included, to that model's provider.
Telemetry
Every compaction emits, in order:
[:nous, :compaction, :start]— measurements%{system_time:, monotonic_time:}, metadata%{agent_name:, messages_before:}[:nous, :compaction, :stop]— measurements%{duration:, bytes_before:, bytes_after:}, metadata%{agent_name:, messages_before:, messages_after:, llm_called:, summarized:, provider:, model:, usage:}[:nous, :compaction, :exception]— measurements%{duration:}, metadata%{agent_name:, messages_before:, messages_after:, llm_called:, kind:, reason:, stacktrace:}
Crash visibility
:compaction_in_progress is set in deps[:summarization_config] before the
:start event and cleared only after :stop. A compaction that dies in the
middle therefore leaves the marker set and an orphaned :start with no
matching :stop, instead of looking like a clean no-op.
Safety
- Never splits tool_call/tool_result pairs
- Keeps system prompt intact
- Falls back to keeping the (pruned) messages if summarization fails