Tool for fetching and extracting readable content from web pages.
Uses Req for HTTP and Floki for HTML parsing. Strips scripts, styles, and navigation to extract the main content.
Dependencies
Requires the floki package in your mix.exs:
{:floki, "~> 0.36"}Usage
agent = Agent.new("openai:gpt-4",
tools: [&WebFetch.fetch_page/2]
)Limits
A fetch buffers at most 5_000_000 bytes and refuses anything whose
content-type is not text/html, application/xhtml+xml, or
text/plain — the URL comes from the model, and the extractor only
understands markup. Move the ceiling with ctx.deps[:web_fetch_max_bytes]
or config :nous, web_fetch_max_bytes: bytes; a max_bytes tool argument
may only lower it, never raise it.
Summary
Functions
Fetch a web page and extract its readable content.
Functions
@spec fetch_page(Nous.RunContext.t(), map()) :: map()
Fetch a web page and extract its readable content.
Arguments
- url: The URL to fetch (required)
- selector: Optional CSS selector to extract specific content
- max_bytes: Optional cap on the response body, in bytes. May only lower
the host-configured ceiling (see
do_fetch/3), never raise it.
Returns
A map with url, title, content, word_count, and fetched_at.