Search-tool

**search-tool: A Deterministic Web Retriever for AI Agents** Most "AI web search" tools are thin wrappers around a paid search API, with a model summarizing whatever comes back. search-tool solves a different, more overlooked half of the problem: once an agent has a URL, how do you turn it into clean, ranked, trustworthy text — for free, with no infrastructure, and without a language model quietly deciding what counts as "relevant"? search-tool is a free, MIT-licensed npm package (`npm i search-tool`) that takes a URL — or up to ten candidate URLs — plus a query, and returns ranked, source-backed passages. No search API, no API key for retrieval, no LLM inside the retriever itself. Everything happens locally and deterministically: run the same URL and query twice, get the same result twice. **How it works, in one call:** 1. Validate the URL and check robots.txt — disallowed sites return a clear "blocked" status rather than being silently ignored. 2. Fetch the page, protected against SSRF, private-network access, redirect loops, and oversized responses. 3. Self-heal broken URLs. Models frequently invent or guess wrong paths. Instead of failing, search-tool checks the site's sitemap and link structure for the closest real match and continues — no round trip back to the model. 4. Parse the content by actual format: HTML, JSON, XML, RSS/Atom, text, Markdown, CSV, and optionally PDF (dependency-free, no native binaries, safe for serverless). 5. Rank the extracted passages locally with BM25 — classical, debuggable, deterministic relevance scoring, not a model's guess. One call is usually enough; the calling model rarely needs to loop. **Why it holds up:** - Zero API keys for the core retrieval path, zero hosted infrastructure, runs anywhere Node or Edge runtimes run. - Deterministic ranking means retrieval itself can't hallucinate — any hallucination has to happen downstream, in generation, not here. - Built-in adapters for OpenAI, Anthropic Claude, and Gemini function-calling, plus a standalone MCP server, all accepting up to 10 candidate URLs with automatic fallback if the first fails. - Domain allow-/deny-lists to restrict or exclude sources, and three retrieval-depth modes (fast/balanced/deep) to trade thoroughness for request budget. **Honest about scope:** it's not a search engine — it reads pages you point it at, it doesn't crawl or index the open web. One site per call. No JavaScript rendering for client-side-only pages. No login/session support. Scanned PDFs return no text without a supplied OCR step. It returns what pages say, not what's true. For this submission, we paired search-tool with Google's Gemini (free-tier flash-lite) in a terminal chat CLI. Gemini is forced to call search-tool's web_retrieve function with candidate URLs and a query; the tool fetches, self-heals if needed, and ranks evidence with BM25; that evidence is fed back to Gemini, which answers only from what was retrieved and cites its source. The CLI renders each stage — tool call, retrieval trace, ranked evidence, grounded answer — so the retrieval mechanics are visible, not hidden behind a single black-box response. The broader point: as AI agents get more autonomy on the web, the tools mediating that access deserve the same rigor as any production infrastructure — predictable behavior, real security boundaries, and honest limits — not a disposable wrapper around a paid API. search-tool shows that classical information retrieval (robots-respecting fetches, sitemap repair, BM25) still earns its keep in an agentic world, installable with one command, running anywhere Node does.