📊 Observability & Telemetry

Nous instruments every layer of agent execution with :telemetry events. You get structured visibility into agent runs, provider API calls, tool executions, the fallback chain, hooks, skills, and workflows — without changing a line of your agent code.

This guide covers attaching the built-in handler, the full event catalog (every name verified against source), writing custom handlers, and exporting Prometheus metrics through the bundled Nous.PromEx.Plugin.

Quick Reference

• Just want logs in dev? → Default Handler
• Need to wire into your own metrics? → Custom Handlers
• Exporting to Prometheus/Grafana? → Prometheus via PromEx
• What events fire and when? → Event Catalog

Default Handler

For local development and debugging, Nous ships a logging handler that attaches to the core agent, provider, tool, context, and callback events:

Nous.Telemetry.attach_default_handler()

It logs at sensible levels:

• Agent runs → info
• Provider requests and stream connections → debug
• Tool executions, iterations, context updates, callbacks → debug
• Tool retries / timeouts → warning
• Agent, provider, and stream exceptions → error

Provider error reasons are summarized (status + a capped body snippet, headers dropped) so a failing upstream cannot flood your logs or leak response context.

Detach it when you're done — for example in tests:

Nous.Telemetry.detach_default_handler()

The default handler attaches the core events only. Fallback, hook, skill, and workflow events have their own handlers or are meant for custom instrumentation — attach to them explicitly (see below).

Event Catalog

All durations are emitted in :native time units. Convert with System.convert_time_unit(duration, :native, :millisecond). The canonical source of truth is the moduledoc of Nous.Telemetry (and Nous.Workflow.Telemetry for workflow payloads).

Agent Events

Provider Events

These live under the [:nous, :provider, ...] namespace (there is no [:nous, :model, ...] event namespace — the PromEx metric names use model, but the underlying events are always provider).

Provider Streaming Events

Tool Events

Context & Callback Events

Fallback Events

Use these to alert on provider degradation. See the workflows and best-practices guides for fallback configuration.

Hook & Skill Events

Workflow Events

Full payloads are documented in Nous.Workflow.Telemetry.

Operational Events

Research observability: the research pipeline (Nous.Research) does not emit its own telemetry namespace. Because it runs ordinary agents internally, you observe it through the [:nous, :agent, ...] and [:nous, :provider, ...] events above.

Custom Handlers

Attach your own handler with :telemetry.attach/4 (single event) or :telemetry.attach_many/4 (many events). The handler receives event_name, measurements, metadata, config:

:telemetry.attach(
  "my-nous-handler",
  [:nous, :agent, :run, :stop],
  fn _event, measurements, metadata, _config ->
    MyApp.Metrics.track_agent_run(
      metadata.agent_name,
      measurements.duration,
      measurements.total_tokens
    )
  end,
  nil
)

Cost & token tracking

Token counts on [:nous, :agent, :run, :stop] make per-run cost tracking trivial. Aggregate into an Agent, ETS table, or your metrics backend:

defmodule MyApp.CostTracker do
  use Agent

  def start_link(_),
    do: Agent.start_link(fn -> %{total_tokens: 0, requests: 0} end, name: __MODULE__)

  def stats, do: Agent.get(__MODULE__, & &1)

  def attach do
    :telemetry.attach(
      "cost-tracker",
      [:nous, :agent, :run, :stop],
      &__MODULE__.handle/4,
      nil
    )
  end

  def handle([:nous, :agent, :run, :stop], measurements, _meta, _config) do
    Agent.update(__MODULE__, fn s ->
      %{s | total_tokens: s.total_tokens + measurements.total_tokens, requests: s.requests + 1}
    end)
  end
end

Prompt-cache tokens

The measurements above are enough to count tokens, but not enough to price them once prompt caching is in play. Cached input is billed at a different rate from fresh input, and the [:nous, :agent, :run, :stop] measurement map carries only total_tokens, input_tokens, output_tokens, tool_calls, requests, and iterations — no cache counters. Those live on the %Nous.Usage{} struct itself (since 0.16.2):

Both default to 0, and Nous.Usage.add/2 sums them like every other counter, so aggregating usage across runs keeps cache accounting intact.

Two provider-shaped caveats decide the arithmetic:

• Anthropic reports the cache counters outside input_tokens, and Nous computes total_tokens as input_tokens + output_tokens for that provider. So on a cache hit total_tokens under-counts what you are billed for; the billed prompt total is input_tokens + cache_creation_input_tokens + cache_read_input_tokens.
• Gemini folds cached tokens into promptTokenCount, so cache_read_input_tokens is a subset of input_tokens. Charging the full input rate on input_tokens over-counts; subtract the cached portion first.
• OpenAI-compatible providers go through Nous.Messages.OpenAI.parse_usage/1, which does not map prompt_tokens_details.cached_tokens — both fields stay 0 there, and OpenAI's cache discount is already folded into prompt_tokens.

That gives one corrected formula with disjoint buckets:

cost = fresh_input_tokens          * input_rate
     + cache_creation_input_tokens * cache_write_rate
     + cache_read_input_tokens     * cache_read_rate
     + output_tokens               * output_rate

where fresh_input_tokens =
  input_tokens                                 # Anthropic (disjoint)
  input_tokens - cache_read_input_tokens       # Gemini (nested)

Cache rates are provider pricing, not derived constants — at time of writing Anthropic charges 1.25x the base input rate for a 5-minute cache write and 0.1x for a read, and Gemini prices cached input well below fresh input while billing cache storage per hour on the CachedContent object rather than per request. Look up the current numbers; the shape is stable.

defmodule MyApp.Cost do
  @moduledoc "Cache-aware cost math over %Nous.Usage{}."

  # USD per 1M tokens. Replace with current provider pricing.
  @rates %{
    anthropic: %{input: 3.00, cache_write: 3.75, cache_read: 0.30, output: 15.00},
    gemini: %{input: 1.25, cache_write: 0.00, cache_read: 0.3125, output: 10.00}
  }

  @doc "Cost in USD for a single `%Nous.Usage{}`."
  def usd(%Nous.Usage{} = usage, provider) do
    rates = Map.fetch!(@rates, provider)
    {fresh_input, cache_write, cache_read} = prompt_buckets(usage, provider)

    millions =
      fresh_input * rates.input +
        cache_write * rates.cache_write +
        cache_read * rates.cache_read +
        usage.output_tokens * rates.output

    millions / 1_000_000
  end

  @doc "Fraction of prompt tokens served from cache, 0.0..1.0."
  def cache_hit_ratio(%Nous.Usage{} = usage, provider) do
    {fresh_input, cache_write, cache_read} = prompt_buckets(usage, provider)

    case fresh_input + cache_write + cache_read do
      0 -> 0.0
      prompt_tokens -> cache_read / prompt_tokens
    end
  end

  # Normalize every provider to three disjoint prompt buckets.
  defp prompt_buckets(usage, :gemini) do
    # Gemini's promptTokenCount already includes the cached tokens.
    fresh = max(usage.input_tokens - usage.cache_read_input_tokens, 0)
    {fresh, usage.cache_creation_input_tokens, usage.cache_read_input_tokens}
  end

  defp prompt_buckets(usage, _anthropic_shaped) do
    # Anthropic reports the cache counters outside input_tokens.
    {usage.input_tokens, usage.cache_creation_input_tokens,
     usage.cache_read_input_tokens}
  end
end

Feed it the usage struct off the run result rather than the telemetry measurements:

{:ok, result} = Nous.run(agent, "Summarize the attached contract")

MyApp.Cost.usd(result.usage, :anthropic)
MyApp.Cost.cache_hit_ratio(result.usage, :anthropic)

Keep telemetry handlers fast and crash-free. A raising handler is detached by :telemetry after one failure, silently dropping your metrics. Do heavy work (DB writes, HTTP calls) off the calling process — hand off to a Task or a queue rather than blocking the agent's hot path.

For a complete, runnable walkthrough — custom handler, the default handler, and a live cost tracker — see examples/advanced/telemetry.exs.

telemetry_metrics

If you already use :telemetry_metrics (e.g. with Phoenix LiveDashboard or a reporter), define metric specs over the event names above:

import Telemetry.Metrics

def metrics do
  [
    counter("nous.agent.run.start.count"),
    distribution("nous.agent.run.stop.duration", unit: {:native, :millisecond}),
    sum("nous.agent.run.stop.total_tokens"),
    counter("nous.tool.execute.stop.count", tags: [:tool_name]),
    counter("nous.tool.timeout.count", tags: [:tool_name])
  ]
end

This plugs into telemetry_metrics_prometheus, telemetry_metrics_statsd, or Phoenix LiveDashboard. For a turnkey Prometheus setup, prefer the PromEx plugin below.

Prometheus via PromEx

Nous bundles Nous.PromEx.Plugin, a PromEx plugin that turns the telemetry events into Prometheus metrics. The plugin module is only compiled when PromEx is present, so it is an opt-in dependency.

Setup

Add PromEx (and its :plug dependency) to your mix.exs:

{:prom_ex, "~> 1.11"},
{:plug, "~> 1.18"}  # required by PromEx

Register the plugin in your PromEx module:

defmodule MyApp.PromEx do
  use PromEx, otp_app: :my_app

  @impl true
  def plugins do
    [
      # ... other plugins
      {Nous.PromEx.Plugin, []}
    ]
  end
end

Options

Nous.PromEx.Plugin implements event_metrics/1 and accepts:

• :otp_app — OTP app name (defaults to the PromEx module's setting)
• :metric_prefix — custom metric prefix (defaults to [otp_app, :nous], or [:nous] when no otp_app)
• :duration_unit — :second, :millisecond (default), :microsecond, or :nanosecond; also selects the histogram bucket scheme

Exposed metric groups

The plugin builds three PromEx.MetricTypes.Event groups:

• :nous_agent_event_metrics — from [:nous, :agent, :run, :stop] and [:nous, :agent, :run, :exception]: run duration, total/input/output tokens, tool-call count, iteration count, and an exceptions counter. Tagged by agent_name, model_provider, model_name.
• :nous_model_event_metrics — from the [:nous, :provider, :request, ...] and [:nous, :provider, :stream, ...] events: request duration, token distributions, request exceptions, stream-connect duration, and stream exceptions. Tagged by provider, model_name (and has_tool_calls for duration).
• :nous_tool_event_metrics — from [:nous, :tool, :execute, :stop] and [:nous, :tool, :execute, :exception]: execution duration, attempt count, and an exceptions counter. Tagged by tool_name, success/will_retry.

Naming caveat: the metric group and metric names use the word model (e.g. nous_model_request_duration), but they are driven by the [:nous, :provider, ...] events. The two are intentionally different — do not try to attach to [:nous, :model, ...]; no such event exists.

Once exposed via PromEx's /metrics endpoint, scrape it with Prometheus and build Grafana dashboards over agent latency, token spend, and tool failure rates.

Related Guides

• Production Best Practices — monitoring, reliability, and deployment patterns
• Workflows — workflow/node telemetry in context
• Hooks — hook lifecycle and the events they emit
• Skills — skill activation/deactivation events
• Troubleshooting — diagnosing failures surfaced by exception events