Nous.Spill behaviour (nous v0.17.1)

Copy Markdown View Source

Content-addressed overflow for oversized tool results.

A 4 MB grep result costs roughly a million tokens of context and is almost never read in full. Spilling writes it somewhere durable, hands the model a short preview plus a locator, and lets it fetch the rest only if it actually needs to.

Shape

{:ok, locator} = Nous.Spill.save_text(config, %{
  owner: "session-abc",
  source: "file_grep",
  suggested_name: "grep-results.txt",
  content: big_text
})

Nous.Spill.retrieval_hint(locator)
#=> "Read it with the file_read tool at /var/…/session-…/a1b2-grep-results.txt"

{:ok, ^big_text} = Nous.Spill.fetch(locator)

Configuration

Per run, in deps — the convention every other pluggable backend in Nous follows (:memory_config, :summarization_config):

Nous.run(agent, prompt,
  deps: %{
    spill_config: %{
      store: Nous.Spill.Local,
      opts: [root: "/var/lib/nous/spill"],
      max_inline_bytes: 65_536
    }
  }
)

Or application-wide, for a deployment that wants it everywhere:

config :nous, :spill, %{store: Nous.Spill.Local, opts: [root: "/var/lib/nous/spill"]}

deps[:spill_config] wins over application config. With neither, spilling is disabled and results are left inline exactly as before — this is opt-in.

Retention

Spilled files persist until the operator deletes them. There is no reaper, by design: a background process deleting content the model may still hold a locator for is a correctness problem disguised as housekeeping, and the right retention policy depends on the deployment (a per-session tmpdir wiped on exit, a nightly cron, an S3 lifecycle rule). Point :root at a directory you are willing to manage.

Summary

Types

attrs()

What to spill. owner scopes the content (a session id); source names the producer (a tool name) for debuggability; suggested_name is advisory — the backend sanitises it.

config()

Resolved configuration: the backend, its options, and the inline byte ceiling.

Callbacks

fetch(t)

Read spilled content back. MUST round-trip save_text/1's bytes exactly.

retrieval_hint(t)

A sentence telling the model how to retrieve this locator with the tools it actually has. The backend owns this string because only the backend knows whether its ids are readable paths, URLs, or opaque keys.

save_text(attrs)

Persist content and return an opaque locator.

Functions

config(ctx)

Resolve the effective spill configuration, or :disabled.

fetch(locator)

Read spilled content back through the backend that wrote it.

maybe_spill(text, opts)

Spill text if it exceeds the inline ceiling, returning the replacement.

retrieval_hint(locator)

The backend's retrieval sentence for this locator.

save_text(map, attrs)

Persist content through the configured backend.

Types

attrs()

@type attrs() :: %{
  owner: String.t(),
  source: String.t(),
  suggested_name: String.t(),
  content: binary()
}

What to spill. owner scopes the content (a session id); source names the producer (a tool name) for debuggability; suggested_name is advisory — the backend sanitises it.

config()

@type config() :: %{store: module(), opts: keyword(), max_inline_bytes: pos_integer()}

Resolved configuration: the backend, its options, and the inline byte ceiling.

Callbacks

fetch(t)

@callback fetch(Nous.Spill.Locator.t()) :: {:ok, binary()} | {:error, term()}

Read spilled content back. MUST round-trip save_text/1's bytes exactly.

retrieval_hint(t)

@callback retrieval_hint(Nous.Spill.Locator.t()) :: String.t()

A sentence telling the model how to retrieve this locator with the tools it actually has. The backend owns this string because only the backend knows whether its ids are readable paths, URLs, or opaque keys.

save_text(attrs)

@callback save_text(attrs()) :: {:ok, Nous.Spill.Locator.t()} | {:error, term()}

Persist content and return an opaque locator.

Functions

config(ctx)

@spec config(map() | nil) :: {:ok, config()} | :disabled

Resolve the effective spill configuration, or :disabled.

ctx may be a Nous.RunContext, a Nous.Agent.Context, a bare deps map, or nil. Precedence: deps[:spill_config], then config :nous, :spill, then disabled.

fetch(locator)

@spec fetch(Nous.Spill.Locator.t()) :: {:ok, binary()} | {:error, term()}

Read spilled content back through the backend that wrote it.

maybe_spill(text, opts)

@spec maybe_spill(
  String.t(),
  keyword()
) :: {:spilled, String.t(), Nous.Spill.Locator.t()} | :inline

Spill text if it exceeds the inline ceiling, returning the replacement.

Returns {:spilled, replacement_text, locator} when the content was written, or :inline when it was small enough, spilling is disabled, or the backend failed.

Three rules are load-bearing:

  • Best effort. A backend error logs and returns :inline. Spilling is an optimisation; it must never turn a successful tool call into a failure.
  • The notice pays for itself. The replacement — preview plus the notice describing the spill plus the newlines joining them — is never larger than max_inline_bytes. The notice is measured against an upper bound of the omitted-byte count before any preview budget is handed out, so the cap cannot be exceeded by the accounting itself. The one exception is inherent: if the notice alone is larger than the cap (a tiny max_inline_bytes against a long locator id), the notice wins and the preview is empty — a replacement the model cannot act on would defeat the point of spilling.
  • Text only. Content that is not valid UTF-8 is left inline: a byte-slice preview of binary content is noise, and the store's contract is text.

retrieval_hint(locator)

@spec retrieval_hint(Nous.Spill.Locator.t()) :: String.t()

The backend's retrieval sentence for this locator.

save_text(map, attrs)

@spec save_text(config(), attrs()) :: {:ok, Nous.Spill.Locator.t()} | {:error, term()}

Persist content through the configured backend.