HTTP Backend Benchmark

Comparison of Nous.HTTP.Backend.Req (default) and Nous.HTTP.Backend.Hackney for non-streaming POST requests.

Two benchmarks are included:

1. Localhost (bench/http_backend.exs) — in-process Plug.Cowboy server. Measures pure client overhead (pool contention, encode/decode, scheduler interaction). No network in the path.
2. Real endpoint (bench/http_backend_real.exs) — OpenRouter free models. Measures what users actually feel: TLS handshake, real network RTT, real LLM response variance.

Both confirm the same conclusion: Req is the right default, with the gap shrinking but never vanishing as response time grows.

Reproducing

# Localhost — pure client overhead. MIX_ENV=test because the in-process
# Plug.Cowboy server reaches the build through Bypass, which is `only: :test`
# (keeping cowboy/ranch off the :dev code path).
MIX_ENV=test mix run bench/http_backend.exs

# Real endpoint — needs creds in env (no secrets land in source).
OPENROUTER_API_KEY=sk-or-... \
  OPENROUTER_MODEL=nvidia/nemotron-3-nano-30b-a3b:free \
  OPENROUTER_MAX_TOKENS=8 \
    mix run bench/http_backend_real.exs

Localhost results

Hardware: Apple M1 Max, 10 cores, 64 GB RAM Runtime: Elixir 1.20-rc.4 / Erlang 28.2 (JIT enabled) Benchee config: 2 s warmup, 10 s measurement, sequential (parallel: 1)

Observations

1. Sequential small requests: Req wins by ~24% on throughput. Both are well under a millisecond; the difference is unlikely to matter for an LLM workload where the network round-trip dominates by 2–3 orders of magnitude.

2. Parallel small requests (50-way): Req wins decisively (~6.5×). Hackney's :default pool serializes connections to a single host under contention — for a real LLM endpoint this is mitigated by higher round-trip latency (the pool drains faster between requests), but on localhost the contention shows up sharply. Apps doing heavy parallel batching against one provider should stay on Req or pre-allocate a larger hackney pool via :hackney_pool.start_pool/2 and pass pool: :my_pool per call.

3. Large bodies (256 KB ×10): Hackney wins by ~19%. Hackney's binary handling for large payloads is more efficient than Req's middleware stack pays for itself.

Real-endpoint results (OpenRouter)

Two free models tested: a fast MoE (Nemotron 3 Nano 30B-A3B, max_tokens=8, ~400ms responses) and a thinking model (Liquid LFM-2.5 1.2B, max_tokens=300, ~1s responses with 100+ reasoning tokens generated internally).

Sequential = 10 paced requests (4s gap). Parallel = 3 batches of 5 concurrent requests (15s gap between batches). Pacing keeps us under typical free-tier 20/min rate caps.

Nemotron 3 Nano 30B-A3B (fast, max_tokens=8)

Within ~10% across the board. At this latency the localhost "Req is 6.5×" finding completely vanishes.

Liquid LFM-2.5 1.2B Thinking (slower, max_tokens=300)

Once responses run longer than a few hundred ms, the parallel gap reappears. Hackney's per-connection gen_server (hackney_conn) does one mailbox hop per chunk read — long responses = more chunks = more hops piling up. Mint (Req's underlying client) is process-less, so chunk count is irrelevant.

Hackney also has consistently worse p95 tails on sequential — cold connection setup is more expensive than Req's pooled-Mint path.

When to switch from the default

Default is Req. Switch to Hackney if:

• You need HTTP/3 — hackney 4 auto-upgrades via Alt-Svc; Req/Finch doesn't yet.
• Your traffic is purely sequential (no parallel batching) and you want to consolidate on one HTTP family across streaming + non-streaming.
• Your provider serves large response bodies (long completions, embedding batches >100 vectors) — Hackney pulls ahead ~15–20% on the 256KB-body localhost scenario.

Stay on Req if (most users):

• You batch parallel requests against one provider — gap can be 1.5–2× slower on Hackney for ~1s+ responses.
• You use Req middleware (Req.Steps, custom step pipelines).
• You want the lowest p95 tails on cold connections.
• You want the most idiomatic Elixir HTTP API.

When to switch streaming to Hackney

Streaming has defaulted to Nous.HTTP.StreamBackend.Req since 0.15.4, matching the non-streaming default. Hackney is the opt-in alternative, not the incumbent.

The non-streaming bench above is about throughput: how fast can you round-trip a request/response. Streaming is about backpressure: can the consumer pace the producer. Both backends bound the consumer, but they bound it differently.

Req (default) — bounded in-flight byte window. Req's :into callback runs in a supervised Task that forwards each chunk to the consuming Stream.resource with send/2. Producer and consumer share an :atomics counter of in-flight bytes: the producer adds byte_size(chunk) before sending, the consumer subtracts it on receipt. Above the 8 MB high-water mark the producer parks in a receive and resumes once the counter falls below the 1 MB low-water mark. The parked producer is Req's :into callback, so parking it stops draining the socket — backpressure propagates all the way to the wire. Resident memory per stream is bounded in bytes rather than in chunk count. A consumer that stays stalled past :backpressure_max_wait_ms (default 30s) aborts the request and the stream yields {:stream_error, %{reason: :backpressure_overflow, inflight_bytes: n}} rather than wedging forever.

Hackney — strict pull. [{:async, :once}] mode: the consumer calls :hackney.stream_next/1 to ask for ONE more chunk, hackney reads ONE chunk off the socket and delivers it. There is no in-flight window because the mailbox never holds more than one chunk — the producer literally cannot run ahead, no matter how slow the consumer is. (Note the tuple form. The legacy [:async, :once] two-atom form silently puts hackney in push mode, because proplists resolves a bare :async as {:async, true}, forfeiting the guarantee.)

Plain Finch.stream/5 gives you neither: its callback is push-based with no window at all, so a fast LLM (Groq at 500 tok/s) feeding a slow consumer grows the mailbox unboundedly. That was the M-12 finding from the 0.15.0 review and the reason streaming ran on hackney before 0.15.4; the Req backend's byte window is what closed the gap.

Switch streaming to Hackney if:

• Your consumer can block for an unbounded time per chunk (LiveView assigns + diff + push under fan-out, persistence-on-every-chunk, slow IO) and you want a hard one-chunk mailbox bound instead of an 8 MB window.
• You would rather see the stream slow down than see it fail with :backpressure_overflow after 30s of a stalled consumer.
• You are already on the Hackney non-streaming backend and want one HTTP family across both paths.

Stay on Req (most users):

• Chunks arrive at token rate (10–100/sec) and are parsed immediately, so the 8 MB window is never approached.
• One HTTP stack across streaming and non-streaming, with hackney left out of the dependency tree entirely.

Trade-off summary:

Opting into the Hackney stream backend

:hackney is declared optional: true, so it is not in your build unless you ask for it. Add it to your app's deps:

{:hackney, "~> 4.0"}

Then select the backend one of three ways, highest precedence first (the resolution chain lives in Nous.Providers.HTTP.stream/4):

# 1. Per-call opt
Nous.Providers.HTTP.stream(url, body, headers,
  stream_backend: Nous.HTTP.StreamBackend.Hackney)

# 2. Environment variable — also accepts "req" or a fully-qualified
#    custom module name such as "MyApp.MyStreamBackend"
export NOUS_HTTP_STREAM_BACKEND=hackney

# 3. App config
config :nous, :http_stream_backend, Nous.HTTP.StreamBackend.Hackney

With none of the three set, the default is Nous.HTTP.StreamBackend.Req. If hackney is selected but the dependency is missing, Nous logs a warning and falls back to the app config / default instead of crashing on the first request.

Configuration (non-streaming)

See Nous.Providers.HTTP.post/4 for the resolution order. Quick recap:

• Per-call: HTTP.post(url, body, headers, backend: Nous.HTTP.Backend.Hackney)
• Env: NOUS_HTTP_BACKEND=hackney
• App config: config :nous, :http_backend, Nous.HTTP.Backend.Hackney

The env var also accepts req, hackney, or any fully-qualified custom backend module (e.g. MyApp.MyHTTPBackend). Custom modules are resolved via String.to_existing_atom/1 with rescue, so unknown values fall back to the app config / default rather than crash.

These knobs cover the non-streaming path only; the streaming path has its own chain, documented in Opting into the Hackney stream backend.