Close what a crash left open in a session log — by appending, never by truncating.
A crash mid-run leaves an orphaned :turn_start: a turn that opened and
never closed. That is the only durable trace of "we died in the middle of
something", and it is what this module looks for. The repair is new events:
- one synthetic
:tool_resultper tool call the turn still owed, each carrying a risk class (see below), and - a
:turn_endwithreason: :interrupted— the one reason no live loop ever emits, so finding it in a log is unambiguous evidence of a crash rather than of an ordinary exit.Nous.AgentRunner.IterationLoopcloses a turn with:completeor:rejectedonly (its@live_turn_end_reasons); a loop that exits{:error, reason}returns no context at all, so nothing downstream could persist a:turn_endfor it.:interruptedis reserved for this module, and covers both the crash and that lost-tail case.
Nothing is deleted and nothing is rewritten. Truncating back to the last clean boundary is the easy repair and it is the wrong one: the turn really did happen, its tool calls really may have fired, and a log that drops the evidence cannot answer the only question a human has after a crash — did that side effect happen?
Recovery is idempotent. Recovering an already-recovered log finds no
orphan (its :turn_start now has a :turn_end) and appends nothing.
Detection keys on the event type only, never on reason or on any risk
metadata, so a persisted blob that came back through JSON with
"interrupted" where an atom used to be cannot change the outcome. Event
types survive a restore through a literal whitelist
(Nous.Agent.Context.deserialize/1), which is what makes them safe to
dispatch on at all.
The risk classes, and how each is derived
A synthetic result carries metadata.risk, one of:
:tool_not_started— the call never reached the executor. It did not run and had no side effects.:tool_outcome_unknown— the call may have run and its result was lost. A human has to check before retrying.
The distinction is what a person actually needs after a crash, so it is read off the log rather than guessed. In order, first match wins:
| # | What the log shows | Class |
|---|---|---|
| 1 | a :tool_call event for this id, and no :tool_result | :tool_outcome_unknown |
| 2 | no :tool_call event for this id, but the turn logged :tool_call for other ids | :tool_not_started |
| 3 | the turn logged no dispatch at all, and the call's enclosing :step_start has a matching :step_end | :tool_not_started |
| 4 | anything else | :tool_outcome_unknown |
Why each rule says what it says:
- 1. A
:tool_callevent is written when the call is handed to the executor. Its presence means dispatch happened, so the tool may well have run — an unknown outcome, never a safe "did not run", even if the enclosing step later closed. - 2. Absence of a
:tool_callevent is only evidence when the log is known to record dispatches. If the turn recorded one for a sibling call, this call's silence means it never got that far. - 3. With no dispatch bookkeeping to read, the step boundary is the next
best fact.
:step_endis emitted even when the step errored, so a step that closed had nothing in flight: a call still owing a result at that point was never dispatched. - 4. The fallback is deliberately the pessimistic one. Wrongly saying
"may have run" costs a human one check; wrongly saying "did not run" is
how a duplicate charge or a second
rm -rfhappens.
Relationship to Nous.Agent.Context.patch_dangling_tool_calls/1
That function stays exactly as it is: the in-memory fast path. It reads
ctx.messages, finds assistant tool calls with no matching tool result
anywhere in the transcript, and appends one generic synthetic result each. It
is turn-blind, which is its strength — it needs no bookkeeping events, so it
works on a v1-seeded log, on a context a user hand-assembled, and on
everything Nous.run/3 produces today.
This module is the durable, turn-aware generalization. It fires only on an
orphaned :turn_start, repairs only what that turn owed, and classifies each
repair. The differences are the point:
- A clean log — every turn closed — is left completely untouched. No synthetic events at all, even if it contains a dangling call, because a closed turn owing a result is not a crash, and inventing history for it would be the fold lying.
- A log with no turn events has no orphan, so recovery is a no-op and
patch_dangling_tool_calls/1is the only repair available.
When both apply, run recovery first. Its classified results satisfy the
owed calls, after which patch_dangling_tool_calls/1 finds nothing dangling
and does nothing. The other order loses the classification: the generic result
lands first and recovery sees the call as already answered.
Where turns come from, and where they do not
Nous.run_stream/3 runs exactly one iteration and is explicitly out of scope
for turns — it emits no :turn_start, so a streamed session can never present
an orphan here, and patch_dangling_tool_calls/1 remains its repair. That is
a deliberate scope decision, not an omission.
Summary
Types
One event recovery wants to append: {type, data}.
How much a lost tool result endangers a human's assumptions.
Anything recovery can read: a context, a log, or a loaded event list.
Types
@type repair() :: {Nous.Session.Event.type(), map()}
One event recovery wants to append: {type, data}.
@type risk() :: :tool_not_started | :tool_outcome_unknown
How much a lost tool result endangers a human's assumptions.
@type source() :: Nous.Agent.Context.t() | Nous.Session.Log.t() | [Nous.Session.Event.t()]
Anything recovery can read: a context, a log, or a loaded event list.
Functions
Whether the log was left mid-turn, i.e. whether a crash needs repairing.
Examples
iex> log = Nous.Session.Log.new()
iex> {:ok, log} = Nous.Session.Log.append(log, :turn_start, %{turn: 1})
iex> Nous.Session.Recovery.interrupted?(log)
true
iex> {:ok, log} = Nous.Session.Log.append(log, :turn_end, %{turn: 1, reason: :complete})
iex> Nous.Session.Recovery.interrupted?(log)
false@spec open_turns(source()) :: [Nous.Session.Event.t()]
The :turn_start events that never got a :turn_end, innermost first.
Turns do not nest, so this is empty or a single event in every log a healthy
emitter produces. It returns a list anyway because a corrupted or
double-started log must be describable rather than crash the load — and
because Nous.Session.fork/2 needs to name the turn a boundary landed inside.
The events recovery would append, in append order, without appending them.
One {:tool_result, data} per owed call, then one {:turn_end, data} per
orphaned turn. Empty for a clean log — which is the whole guarantee that
recovery never touches a session that did not crash.
Exposed because "what would you change, and why" is a question an operator asks before accepting a repair, and because a caller holding neither a log nor a context (a raw event list straight off a persistence backend) can still inspect the damage.
@spec recover(Nous.Session.Log.t()) :: Nous.Session.Log.t()
@spec recover(Nous.Agent.Context.t()) :: Nous.Agent.Context.t()
Apply plan/1, appending the repairs.
Returns the same kind it was given. A Nous.Session.Log gets the events
appended directly; a Nous.Agent.Context gets them through add_message/3
and log_event/3 so messages and the fold stay in lockstep — a synthetic
tool result is a surface event, and writing it any other way would leave the
two disagreeing.
A clean log comes back untouched, including its fold cache: plan/1 is empty,
so nothing is appended and nothing re-materializes.
needs_response is deliberately preserved on the context path, matching
patch_dangling_tool_calls/1. Appending a tool result normally flips it to
true, which would make repairing a crashed session equivalent to deciding to
resume it. Recovery's job is to make history honest; whether to run again is
the caller's call.