Nous.ModelDispatcher (nous v0.17.1)

Dispatches model requests to the appropriate provider implementation.

Routes requests to providers based on the model's provider field:

:anthropic → Nous.Providers.Anthropic
:gemini → Nous.Providers.Gemini
:vertex_ai → Nous.Providers.VertexAI
:mistral → Nous.Providers.Mistral
:lmstudio → Nous.Providers.LMStudio
:llamacpp → Nous.Providers.LlamaCpp
:vllm → Nous.Providers.VLLM
:sglang → Nous.Providers.SGLang
:openai → Nous.Providers.OpenAI
:custom → Nous.Providers.Custom
Others → Nous.Providers.OpenAICompatible

Swapping the dispatcher

Every model request in Nous goes through a resolved dispatcher module rather than through this module directly. See resolve/1 for the precedence rules, config :nous, :model_dispatcher, MyDispatcher for the production injection point, and put_dispatcher/1 for the process-scoped testing seam.

Summary

Functions

count_tokens(model, messages)

Count tokens (uses appropriate provider implementation).

provider_module(provider)

Resolve the provider module for a provider atom.

put_dispatcher(module)

Install a process-scoped dispatcher override. Testing seam only.

request(model, messages, settings)

Dispatch request to the appropriate provider implementation.

request_stream(model, messages, settings)

Dispatch streaming request to the appropriate provider implementation.

resolve(override \\ nil)

Resolve the module that services model requests, highest precedence first

Functions

count_tokens(model, messages)

@spec count_tokens(Nous.Model.t(), list()) :: integer()

Count tokens (uses appropriate provider implementation).

provider_module(provider)

@spec provider_module(atom()) :: module()

Resolve the provider module for a provider atom.

Unknown providers fall back to Nous.Providers.OpenAICompatible.

put_dispatcher(module)

@spec put_dispatcher(module() | nil) :: :ok

Install a process-scoped dispatcher override. Testing seam only.

Unlike config :nous, :model_dispatcher, MyDispatcher — which remains the production mechanism — this writes no global state. The module is stored in the calling process's dictionary and is visible to that process and to any process that lists it in $callers; Task.async/1, Task.async_stream/3 and Task.Supervisor.async/2 all propagate $callers, so work the runner fans out still sees the override. Concurrent (async: true) test modules can therefore each install their own stub without contending for the application environment.

The override lives and dies with the calling process. ExUnit gives every test a fresh one, so there is nothing to tear down; pass nil to clear it early.

$callers is not propagated across GenServer.start_link/3, so a run driven through Nous.AgentServer executes in the server process and will not see the override. Those tests still need the application environment, and must stay async: false because of it.

test "retries on a provider error" do
  Nous.ModelDispatcher.put_dispatcher(FlakyDispatcher)
  assert {:ok, _} = Nous.generate_text("openai:gpt-4", "hi")
end

request(model, messages, settings)

@spec request(Nous.Model.t(), list(), map()) :: {:ok, map()} | {:error, term()}

Dispatch request to the appropriate provider implementation.

request_stream(model, messages, settings)

@spec request_stream(Nous.Model.t(), list(), map()) ::
  {:ok, Enumerable.t()} | {:error, term()}

Dispatch streaming request to the appropriate provider implementation.

resolve(override \\ nil)

@spec resolve(module() | nil) :: module()

Resolve the module that services model requests, highest precedence first:

override — an explicit per-call module, such as Nous.LLM's :model_dispatcher option. nil means "not specified".
A process-scoped override from put_dispatcher/1, searched in the calling process and then along $callers.
config :nous, :model_dispatcher, MyDispatcher.
Nous.ModelDispatcher.