GroktoCrawl 1.0: Give Agents the Whole Evidence Trail

Ask an autonomous agent to research a complex technical question, and it will almost always hand you a confident, well-formatted summary. Ask that same agent to prove exactly which sentence in a forty-page vendor specification justified its conclusion, and the illusion instantly collapses.

In most modern AI development stacks, web research is treated as an ephemeral prompt wrapper: make an API call, scrape a few URLs, grab the first four kilobytes of text, dump the resulting pile of tokens into a context window, and ask a model to synthesize the answer. When the task is trivial, that shallow approach works well enough. When the task carries real operational stakes; evaluating a million-dollar software proposal, auditing a compliance framework, or tracing a security vulnerability across upstream open-source code; it fails in ways that are quiet, pervasive, and dangerous.

The failure modes are well known to anyone who has run agents against real enterprise documents. Models play telephone, summarizing summaries of summaries until critical qualifications vanish into thin air. Ephemeral scrapers suffer from opening-slice blindness, happily evaluating the introductory paragraphs of a document while remaining completely oblivious to the crucial liability exclusion buried on page fourteen. Worse, agents generate unanchored citations; pointing to a top-level domain or an entire whitepaper without a page number, character offset, or direct quotation. When an executive or engineer attempts to audit the claim, there is no evidence trail to inspect.

GroktoCrawl 1.0 was designed to dismantle that brittle paradigm. What started as a self-hosted, Firecrawl-compatible web-data scraper has evolved into something far more ambitious: an experiment-grounded, sovereign research substrate engineered to give both humans and autonomous agents a complete, auditable chain of custody from raw web capture down to the exact cited character offset.

From ephemeral scraper to sovereign research substrate

The leap from version 0.14 to GroktoCrawl 1.0 isn't a routine version bump or a cosmetic interface refresh. It represents the unification of the mainline project with the extensive research architecture developed under GroktoCrawl X, an ambitious overhaul spanning over 400,000 lines of code, 1,900 files, and more than thirty formal Architecture Decision Records (ADRs). Rather than treating web search and scraping as throwaway utility steps, GroktoCrawl 1.0 treats research as a stateful, auditable software pipeline.

The core philosophy is straightforward: an autonomous agent should never be asked to evaluate claims in the dark, and a human should never be expected to trust an agent's conclusion without seeing the exact evidence that produced it. To make that real in production, the system had to solve five interrelated technical challenges: eliminating document truncation through full-source lexical selection, allowing private organizational documents to be evaluated directly alongside live web crawls, filtering web noise with continuous semantic evaluation, unifying storage in a sovereign PostgreSQL backend, and enforcing deterministic operational bounds with restart recovery.

The result is a self-hosted, MIT-licensed platform that runs entirely within your own infrastructure; backed by PostgreSQL, pgvector, Valkey, and headless Playwright browsers; exposing seventy-three Model Context Protocol (MCP) tools, an audited command-line interface, and an enterprise HTTP API.

Eliminating the opening-slice trap

The most insidious defect in automated document extraction is opening-slice truncation. Traditional AI scraping pipelines read the first 8,000 or 16,000 characters of an acquired HTML page or PDF, truncate the remainder to conserve token budgets, and feed the stump to the model. If the vendor's actual service-level guarantee or data-retention exemption sits after the 25,000-character mark, it simply doesn't exist as far as the agent is concerned.

GroktoCrawl 1.0 replaces arbitrary truncation with full-source bounded lexical selection. When GroktoCrawl captures a document; whether it's a sprawling technical specification, a legislative bill, or a multi-section API manual; it acquires the complete text, computes cryptographic source hashes across the body, and scans the entire document to extract relevant evidence passages based on the research objective.

Instead of feeding the model a truncated preamble, GroktoCrawl extracts targeted, bounded passages from anywhere across the document body, complete with exact character offsets (such as characters 14,210 through 14,880). The API response surfaces the source hash, the exact character boundaries selected, and a precise coverage metric indicating how much non-relevant text was omitted. Callers can enforce explicit evidence budgets, tuning the depth of material retrieved.

For an engineering team or compliance auditor, this changes the nature of automated review. When the agent cites an indemnification clause or an API rate-limit exception, the citation isn't an unanchored link to a forty-page manual. It's a resolvable pointer to the exact character offset in the acquired document. A human reviewer can navigate directly from an executive summary down to the raw captured text, verifying the surrounding context without asking the model to paraphrase its own memory.

This establishes a practical version of the artifact pyramid: start with an executive summary, and provide an unbroken route down through structured analysis to the underlying source material whenever you need to inspect the reasoning. A human reviewer doesn't need every page in front of them at once. They need a verifiable path to the evidence that matters.

Putting private documents beside the public record

High-stakes research rarely happens entirely on the public internet. In real organizations, the most critical evaluations require putting proprietary, confidential documents right beside the public record. You have an NDA-protected vendor proposal on your desk, and you need to compare its architectural promises against the vendor's public developer documentation, open-source issue trackers, and customer outage reports.

Historically, teams had to choose between two bad options: paste sensitive internal proposals into commercial search engines, or run isolated local document Q&A tools that had zero awareness of live web realities. GroktoCrawl 1.0 breaks down that artificial wall through owned research sessions.

Using GroktoCrawl's local-first Parse service, teams can upload internal PDFs, Word documents, or technical memos directly into an owned, expiring session. The document is parsed locally, its text and structural headers are extracted into isolated storage, and it gets an authenticated source identifier. An agent can then query across selected private document attachments and live web crawls simultaneously.

The reviewer can now ask: does the uptime commitment in Section 4 of the proposal match the failover architecture described in the vendor's public documentation? GroktoCrawl returns answers grounded in citations drawn from both worlds: citing the private proposal's exact character offsets on one side, and the live public web captures on the other. Crucially, private documents follow strict session expiration and deletion rules; they retain extracted text for the active investigation rather than becoming a permanent, unmanaged data lake.

Surgical noise filtration: scored material contribution with Jev

Even when a web crawler successfully retrieves an HTTP 200 response, the resulting payload is often packed with digital garbage. Soft 404 error pages, cookie consent banners, robot challenge landing screens, and sprawling header navigation bars all report successful status codes while contributing precisely zero useful information to the user's research objective. Dumping that noise into a language model's context window wastes expensive tokens, distracts synthesis, and invites subtle hallucinations.

To eliminate that junk before synthesis begins, GroktoCrawl 1.0 introduces an optional, post-scrape noise filtration layer powered by Jev (described in Groktopus's analysis of semantic decision models). Rather than relying on fragile heuristic regexes or blunt character counts, Jev evaluates the material contribution of newly acquired page text against the user's explicit research objective.

When configured, Jev assesses the entire acquired page in overlapping semantic chunks and assigns a continuous material_contribution_score from 0.00 to 1.00. Because a single relevant section is enough to justify reading a page, the page's final score is the maximum score of its chunks. Under frozen benchmark replays, obvious non-content pages scored between 0.02 and 0.06, while substantive technical articles scored substantially higher.

By enforcing a conservative contribution threshold (omitting pages scoring below 0.10 from synthesis), GroktoCrawl purges non-content noise before it reaches the language model. The raw score remains recorded in the source metadata for full auditability, while private and local network sources are strictly protected from external evaluation. The result is a cleaner, higher-density context window that feeds the synthesis engine only the evidence that actually bears on the question.

The empirical discipline: why 540 trials killed adaptive querying

In the generative AI landscape, the prevailing instinct is to add agentic complexity at every turn: recursive planning loops, dynamic query generation, and autonomous tool fanout. When building GroktoCrawl 1.0, the engineering team took the opposite path: every major architectural hypothesis had to survive frozen empirical trials before earning a spot in the codebase.

The clearest example of this discipline was the W10 adaptive query planning benchmark. The team spent weeks designing an intelligent "adaptive query runtime" that allowed an agent to inspect initial search results, dynamically invent follow-up query angles, and branch out across the web autonomously. On paper, it was the quintessential modern AI agent feature.

To evaluate whether the feature actually improved research quality, the team froze a benchmark protocol across 540 controlled, empirical trials, testing anchor cases and adversarial challenge cases against strict replacement gates. The results were humbling: dynamic query fanout repeatedly introduced query drift, exploded API token costs, and failed to reliably outperform disciplined, deterministic multi-query retrieval on repeatability and claim accuracy.

Rather than shipping a flashy, unpredictable agentic feature, the team made the disciplined engineering decision: they rejected adaptive query fanout and kept fixed-query retrieval as the production default. In autonomous systems, piling on unproven complexity isn't a sign of ambition; it's just technical debt waiting to detonate in production.

That same experimental rigor governed every layer of the 1.0 architecture:

• Claim verification experiments (W12.3): Validating that models verify factual claims against verified passage extractions before committing them to final report artifacts.
• Post-scrape noise filtration (Cardiac JEV Replay): Running replay evaluations on dense web scrapes to prove that noise filters strip boilerplate without corrupting financial tables, legal footnotes, or code snippets.
• Evidence obligation replays: Ensuring that when an agent claims a topic has been researched, it accounts for every planned query and source candidate rather than declaring victory after three easy hits.

Consolidating storage sovereignty: the PostgreSQL and pgvector migration

A research system that depends on a sprawling constellation of specialized cloud datastores is neither sovereign nor simple to operate. Earlier iterations of GroktoCrawl relied on Qdrant as a mandatory vector storage dependency, forcing operators to maintain separate database clusters, distinct backup runbooks, and uncoordinated recovery mechanisms.

GroktoCrawl 1.0 achieves storage sovereignty by consolidating relational records, durable research ledgers, and semantic vector embeddings directly into PostgreSQL using the pgvector extension. Under PostgreSQL Schema 16, teams can run GroktoCrawl in a dedicated pgvector_only mode, managing their entire operational footprint; from crawl metadata and session state to dense embeddings; within a single, battle-tested ACID database engine.

True infrastructure maturity, however, lies in how migrations are handled. GroktoCrawl did not force an abrupt cutover. Instead, it engineered a fully verified, reversible migration path backed by ADR-0071. Operators can run in shadow_pgvector mode, where live reads continue serving reliably from Qdrant while new embeddings are mirrored into PostgreSQL. Idempotent backfill scripts copy historical vectors, execute rigorous numerical parity checks, and allow operators to verify vector alignment before promoting PostgreSQL to read authority.

If an unexpected operational issue arises, the system includes documented, tested reverse-copy runbooks to restore Qdrant without losing newly acquired research state. By treating vector storage as an integrated capability of PostgreSQL rather than an exotic separate cluster, GroktoCrawl makes enterprise deployment radically simpler to secure, back up, and scale.

Bounded substrates, operation receipts, and crash recovery

An autonomous research agent running without hard operational boundaries is a financial and architectural hazard. An agent that encounters an ambiguous query can easily burn through hundreds of dollars in external API credits in an afternoon, or lock up worker threads in endless retry loops when an upstream source throttles its requests.

GroktoCrawl 1.0 enforces hard operational ceilings across its durable research engine. Every research run operates under strict, immutable budgets: capped query counts (typically three planned queries), a maximum of eight retained sources, a 256-kilobyte raw source ceiling, sixty-four model invocations, and a 180-second hard timeout from admission.

Equally critical is how the system handles failure. In distributed systems, networks drop and containers restart. If an external model or scraping call is interrupted mid-flight, naive agents blindly repeat the request; racking up duplicate API charges and polluting databases with redundant records. GroktoCrawl implements persistent operation receipts. When an external call completes, its cryptographic receipt is committed to durable storage. If a worker crashes and restarts, it inspects its receipt ledger; if a call was dispatched but its outcome cannot be verified, it halts and reports an explicit outcome_unknown status instead of guessing.

Furthermore, research investigations in GroktoCrawl are never dead-end chat transcripts. Once an investigation finishes, its verified evidence is preserved as a durable research artifact. An analyst or agent can preview an explicit follow-up question, select the exact retained references to carry forward, and branch into a new, bounded investigation without having to re-crawl or re-scrape the original sources.

Giving agents real tools through MCP and CLI

For an AI agent, software is only as good as its tool interfaces. Handing an agent a generic bash shell or an unstructured web endpoint turns tool execution into a guessing game. GroktoCrawl 1.0 ships with native, typed integration across seventy-three Model Context Protocol (MCP) tools alongside a unified command-line interface.

The MCP surface turns autonomous agents into structured research operators. An agent running in Claude Desktop, Hermes, or an autonomous workflow engine can:

• Initialize an owned research session and attach local document bytes.
• Generate and inspect a research plan before requesting human approval for execution.
• Execute targeted full-source scrapes with automatic challenge barrier recovery.
• Resolve character-offset citations into full retained text.
• Store, query, and sweep durable research memory across multiple sessions.
• Preview and launch explicit contextual follow-ups anchored in verified evidence.

By exposing identical capabilities across HTTP, the command line, and MCP, GroktoCrawl ensures that an engineer debugging an investigation in their terminal, a developer integrating research into a backend service, and an autonomous coding agent executing a multi-step workflow are all driving the exact same audited substrate.

What true research autonomy demands

The release of GroktoCrawl 1.0 brings a vital clarification to enterprise AI: autonomy isn't about how fluently an agent generates prose. True autonomy is about whether the agent's work can be audited, verified, and defended when real decisions are on the line.

When an agent presents an executive with a strategic recommendation, the value doesn't lie in the eloquence of the summary. The value lies in the integrity of the evidence chain: knowing that the agent scanned the complete document rather than the opening slice; knowing that its citations resolve to exact character ranges rather than unanchored homepages; and knowing that its operational bounds were proven through hundreds of empirical trials rather than uncalibrated enthusiasm.

By anchoring autonomous research in full-source lexical selection, private-document integration, Jev-scored noise filtration, sovereign PostgreSQL storage, durable operation receipts, and bounded execution, GroktoCrawl 1.0 shows what earned autonomy actually looks like in practice. It turns AI research from an ephemeral parlor trick into an auditable evidence substrate; one where every claim carries its receipts, and every conclusion can be verified before an organization stakes a decision on it.