LocalAgentBrowser · Technical paper

Verification over Trust:Code-Enforced Grounding for Honest Reporting by Small Local Language Models in Autonomous Web Research

SnowFX · LocalAgentBrowser 1.63 · November 2026

Update, November 8, 2026: this paper reports version 1.63. Version 1.64 reran the four-model round on identical code with a 90-minute limit: Qwen2.5-7B finished 6 of 7 tasks, Llama-3.1-8B 7 of 7, Qwen2.5-14B 5 of 7 and Qwen2.5-3B 3 of 7, with no finished result filling a gap from the model's own knowledge. One finished 3B result contains a claim the measures missed (a misread link card). Details: regression history. Version 1.65 adds a Windows installer and optional cloud models through OpenRouter; the tests in this paper used local models only. November 16: version 1.66 was measured on four cloud models (Phi-4, GPT-4o-mini, Llama 3.1 8B, Gemma 3 4B), each finishing 7 of 7, and on 50 SimpleQA questions, where Phi-4 alone answered 1 and with LocalAgentBrowser 45; the local models have not been re-run on 1.66 yet.

Abstract

Language models used as research assistants produce fluent answers whose claims are often not supported by any source. They are also unreliable judges of their own work. We describe LocalAgentBrowser, an open-source autonomous web research agent built for a small model (Qwen2.5-7B-Instruct, 4-bit, CPU only, 8,192-token context), in which honesty is enforced by code rather than requested from the model. The model chooses one of five actions per turn. Every finding it records must be copied from the page shown, and deterministic checks verify that before the finding counts. Commands and paths must match the page exactly. A checklist of the objective's parts is fixed at the start. The agent is sent back only for gaps the code can verify. The final answer is written in a separate call from the recorded findings alone, and the code appends what was not found.

We evaluate on a fixed suite of seven task types run end to end on the live web. On identical code, four models (Qwen2.5 3B, 7B and 14B; Llama 3.1 8B) produced 10 finished results, none of which filled a gap from the model's own knowledge or stated a number or command absent from its findings, by the suite's lexical measures. The 7B finished 6 of 7 tasks under a 90-minute limit and 7 of 7 without one. We report the measures' blind spots as findings in their own right: an invented command the scorer counted as clean, an uncited fill, and answers that silently omitted part of the objective. Each run is saved, and the suite can be rerun.


1. Introduction

Ask a language model a research question and it answers in the voice of someone who has checked. Often nobody has. The model may have filled a gap from its training data, mixed sources, or invented a plausible detail. When challenged, many assistants apologise and promise to do better, without any mechanism that makes the next answer more reliable.

A common response is to ask the model to check itself: to cite, reflect, verify or critique. That still relies on the model's own judgement, which is the thing in doubt. This paper takes the opposite position. The model proposes; code decides what counts. We do not try to make a small model wiser. We make it hard for any model's claims to reach the final answer unless code can tie them to a page it actually read.

We built this for the hardest practical setting we could run: a 7-billion-parameter model on an ordinary office PC (Intel i5-9500, six cores, no GPU), where each model call takes from under a minute to several minutes. Small models on a CPU drift, loop, hallucinate and get stuck, and everything they do costs time. This makes their failures easy to see and expensive to ignore, which makes them a good stress test for verification by code.

Contributions.

  1. A design for an autonomous web research agent in which honesty properties are enforced in code: findings grounded in the page shown, with exact-text checks for commands and paths; a fixed objective checklist; code-verified coverage before finishing; and a final result written only from recorded findings, with gaps and caveats added by the code (Section 3).
  2. A reproducible, honesty-focused regression suite of seven task types, with measures of what finished results claim beyond their findings (Section 4).
  3. A cross-model evaluation on identical code (four models), plus an untimed control run (Section 5).
  4. Negative results and measured dead ends, including where our own measures failed (Sections 5.5 and 6).

2. Related work

Most work on making language-model agents reliable improves the model: it trains, fine-tunes or prompts the model to cite, to reflect, or to act better. We take the other route, verification instead of training. We change nothing in the model, and we move the checks out of the model and into code. We organise the related work around that difference.

Attribution and quoting. Retrieval-augmented generation (Lewis et al., 2020) conditions generation on retrieved text. Work on attribution asks whether generated statements are supported by a source: the AIS framework (Rashkin et al., 2021) and citation benchmarks such as ALCE (Gao et al., 2023). These measure attribution after the fact; our agent enforces it when a finding is recorded. The closest precedent for our core rule is GopherCite (Menick et al., 2022), which trained a 280B model to support its answers with verbatim quotes, judged by human raters. Our agent requires the same kind of quote from a 7B model without training, and code, not people, checks it against the page. Verbatim RAG, an open-source project, likewise builds answers only from extracted spans, for document question answering.

Hallucination, factuality and abstention. Surveys of hallucination (Ji et al., 2023) and fine-grained factuality scoring such as FActScore (Min et al., 2023) and SelfCheckGPT (Manakul et al., 2023) supply the measures this field uses; MiniCheck (Tang et al., 2024) shows that a 770M checker can judge support against a document about as well as GPT-4, a natural upgrade for our lexical measures (Section 5.5). The agent's "not found" behaviour is a form of abstention (Wen et al., 2025). ToolBeHonest (Zhang et al., 2024) found that tool-using models often fail to recognise unsolvable tasks and that larger models were not reliably better, which agrees with our finding that honesty did not scale with size (Section 5.2).

Self-correction. Self-Refine (Madaan et al., 2023), Reflexion (Shinn et al., 2023) and Chain-of-Verification (Dhuliawala et al., 2023) let a model revise its own output, and Self-RAG (Asai et al., 2023) trains 7B and 13B models to judge whether their own output is supported. Huang et al. (2024) found that models often cannot correct their reasoning without external feedback. Our measurements agree for a small model: about half of the 7B's own "this item is not covered" verdicts were wrong (Section 3.4). We therefore remove self-grading rather than train it.

Web agents. WebGPT (Nakano et al., 2021) trained a model to browse and cite. ReAct (Yao et al., 2023) interleaves reasoning and actions. Benchmarks include WebArena (Zhou et al., 2024) and Mind2Web (Deng et al., 2023). Open-source research tools such as GPT Researcher and Local Deep Research build multi-step, cited reports, and browser agents such as browser-use mainly target large hosted models. AgentBench (Liu et al., 2024) found a wide gap between commercial models and open models of 70B parameters or fewer.

Small and local models as agents. Most work on small agents closes that gap by changing the weights. AgentTuning (Zeng et al., 2023), FireAct (Chen et al., 2023), Lumos (Yin et al., 2023) and α-UMi (Shen et al., 2024) fine-tune open models on agent trajectories, the last splitting planning, calling and summarising across several small models. WebRL (Qi et al., 2025) uses online reinforcement learning to lift Llama-3.1-8B from 4.8% to 42.4% on WebArena-Lite. Search-R1 (Jin et al., 2025) trains Qwen2.5-3B and 7B, the family we use, to call a search engine while reasoning, and DeepResearcher (Zheng et al., 2025) trains research agents with reinforcement learning on the live web. On-device work such as TinyAgent (Erdogan et al., 2024) and Octopus v2 (Chen and Li, 2024) fine-tunes 1–7B models for fast local function calling. ACBench (Dong et al., 2025) measures what quantisation and pruning cost an agent; our model runs at 4 bits. Belcak et al. (2025) argue that small models should do most agent work, chiefly on cost. Our approach is orthogonal to all of these. We change nothing in the model, and one code path serves every model we tested (Section 3.6), so trying a new model needs no code change. Training can make an agent more capable; it cannot make the agent's claims checkable. The two compose: a WebRL- or Search-R1-trained model could run under our checks, gaining capability from training and verifiability from code.

We are not aware of a published evaluation of a 7B-class model doing open-web research on a CPU-only computer, scored on whether its answers fill gaps from the model's own knowledge. The trained research agents use GPUs and report accuracy; the on-device work concerns tool calling; GopherCite's quotes were rated by people. Our system differs from most of this work in where the check sits: in deterministic code, at the moment a claim is recorded and again when the answer is written, rather than in a model's judgement after the fact. The model never grades its own work.

3. System design

The agent is about 2,500 lines of Python in ten modules, with 251 offline tests that replace the model and the browser with scripted fakes. LM Studio serves the model through an OpenAI-compatible API, and Playwright drives a visible Chromium window.

3.1 Loop and actions

Each turn, the model sees the objective, its numbered findings, a compact journal of recent actions, and the current page part. It replies with one JSON action, constrained by a JSON schema:

Action Effect
SEARCH DuckDuckGo (HTML) results with their real URLs
NAVIGATE Open a URL from the results or the current page
READ_MORE Next part of a long page (parts of about 2,500 characters), or the unread part whose outline label names a missing item
DISCOVER Record a finding from the current page
FINISH End the run, after the checks in Section 3.4

The agent only reads. It does not click, fill in forms or sign in.

A browsing reply is capped at 600 tokens. A 3B model once wrote more than 2,500 tokens in one reply, at about 4 tokens per second. On a CPU, reading the prompt takes about 90% of each call (about 8 prompt tokens per second on our machine). So the agent keeps one growing conversation, which lets LM Studio reuse its prompt cache. It starts a fresh prompt only near 75% of the context budget.

A refused action is corrected inside the same turn ("cheap retry"): a short note is appended, explaining why and what to do instead. This is how every check below talks to the model.

3.2 Grounding: a finding must be on the page shown

A DISCOVER is checked against the text of the page parts shown so far, in order:

  1. Word-pair overlap. If fewer than 70% of the finding's word pairs appear in the page text, or any sentence falls below 35%, the model is asked once to confirm, using only the page's words. Below 40% overall, or 25% for any sentence, the finding is refused outright. Short findings and sentences (fewer than 20 or 8 pairs) are not scored, because a few connecting words would sink their score.
  2. Pointing the way. A refusal quotes the one page sentence closest to what the model wrote, or names what the parts shown never mention ("Japan, Navajo"), so that the model can record the real text or read on. Sentences in which a page describes itself, and questions, are not quoted.
  3. Exact text for code. Commands and paths must appear on the page as written. This covers text in backticks, every line of a fenced code block, and any token with a backslash, a <placeholder> or two or more slashes; URLs and prose such as "macOS/Linux" are excluded. Case, whitespace, quote style and placeholder brackets are normalised. Section 5.4 explains why word overlap cannot catch a wrong command.
  4. Recorded from the wrong place. A finding is refused, with the next step, if it is made only of search-result snippets, the page's headings, the page describing itself ("this guide covers…"), questions, an echo of the objective or of the model's own searches (50% overlap with the objective and less with the page), or the checklist's own wording. Each rule was added after a model did exactly that. Where possible, a rule was checked against the saved findings before it was adopted; the hits found there were junk.

3.3 A fixed objective checklist

At the start, one short call splits the objective into at most eight checklist items, keeping conditions such as "free, open-source" or "three". The checklist is fixed for the run. Every finding names the items it serves, and every later check goes through these items.

Before version 1.53, the model re-split the objective from memory at each check. It drifted: it once blocked its own finish over "migration strategies", which the objective never asked for. Fixing the checklist was the largest single improvement in the project's history (Section 5.1).

3.4 Coverage: verified in code, not taken on trust

When the model chooses FINISH, each checklist item is checked in a short call of its own. The model gives a verdict and the evidence: the numbers of the findings, or words copied from them. The code then verifies the verdict:

Only gaps the code can verify send the agent back: an item nothing was copied for, fewer things than asked for, or nothing recorded at all. The model's other verdicts only shape the final result. This rule comes from measurement. Over the first four suite runs with a coverage check (versions 1.54 to 1.56), about half of the model's send-backs were for gaps that were not real. Examples include "missing official documentation" on a task that did not ask for it, and "H2O, O2" already recorded. Each wrong send-back cost a finish or 8 to 10 minutes.

3.5 Writing the final result

After FINISH, one more call writes the result from only the objective, the checklist and the recorded findings, with short rules: cite findings by number, add nothing from outside, and give as many items as asked for. The code then does what the model cannot be trusted to do:

The report always lists every finding with its source URL, however the run ends.

3.6 One code path for every model

Every rule applies to every model. There are no per-model prompts, branches or profiles. When a new model exposed a failure (Section 5.2), the fix went into the shared code and was measured on the default model too.

4. Evaluation method

4.1 The suite

Seven objectives, one of each task type:

Task Objective (abridged) What it tests
Factual Tallest mountain in Africa, height in meters and feet Basic retrieval and finishing
List Three free, open-source alternatives to Microsoft Office, and what each is best at Counts and conditions
Comparison SQLite vs PostgreSQL for a small web app, then a recommendation Multiple items and reporting a gap
How-to Cold brew coffee: equipment, steps with amounts, how long it keeps Amounts and step completeness
Explanation How photosynthesis works: inputs, outputs, both stages and where Multi-part coverage on long pages
Opinion How at least three cultures have viewed owls, then an opinion Counting, and an opinion kept separate
Documentation From the official Python docs: create, activate (Windows; macOS/Linux) and install into a venv Source conditions and exact commands

4.2 Measures

For each task: finished or not, turns, model minutes, and how many checklist items have findings. For each finished result, the suite counts:

All of these measures are lexical. They detect certain patterns, and they cannot judge meaning. Section 5.5 gives their known misses.

4.3 Setup

4.4 Process rules

4.5 Replays

Many decisions were tested on a single saved moment instead of a whole suite. The exact prompt from a run was sent to the model 3 to 9 times, before and after a change. This tests a change on the moment it is meant for, in minutes rather than hours.

4.6 Scale

From 2026-10-16 to 2026-10-29 we ran 44 suite runs on one machine. They comprised 205 task runs, recorded 525 findings and took 118.7 hours of model time. Every run folder keeps the prompts, the model's replies, the journal and the report (Figure 1).

The testing behind the claims: 44 full test-suite runs, 205 tasks, 525 findings checked against their page, 118.7 hours of the AI working, on one office PC from October 16 to 29, 2026.
Figure 1. The testing behind the claims, 2026-10-16 to 2026-10-29: 44 suite runs, 205 task runs, 525 findings checked against their page, 118.7 hours of model time, one PC.

5. Results

5.1 Progress over versions (Qwen2.5-7B)

Version 1.52 1.53 1.54 1.54b 1.55 1.56 1.57 1.58 1.59 1.60 1.61 1.62 1.63
Finished (of 7) 2 5 4 5 5 4 4 5 5 6 6 7 6
Model minutes 254 200 239 225 233 272 270 214 209.5 170.6 222.1 168.3 197.4
Tasks finished out of 7 by the default 7B model, version by version: 2, 5, 4, 5, 5, 4, 4, 5, 5, 6, 6, 7, 6, and 7 of 7 for version 1.63 with no time limit.
Figure 2. Tasks finished out of 7 by Qwen2.5-7B, version by version (the table above), and 7 of 7 for 1.63 without a time limit (Section 5.3).

Version 1.62 changed no behaviour (a code reorganisation verified by golden-prompt tests), so its 7 of 7 against 1.61's 6 of 7 is run-to-run variance. One run per version cannot separate small effects from variance, and we do not claim it does.

The large steps were the fixed checklist (1.53: 2 → 5) and sending the agent back only for gaps the code verifies (1.58). Several versions finished fewer tasks because they were stricter: a check that refuses more costs turns, and each turn costs minutes.

5.2 Four models, identical code

Four models on the same code: Qwen 2.5 7B finished 6 of 7 tasks, Qwen 2.5 14B 2 of 7, Llama 3.1 8B 2 of 7, each with 0 gaps filled from its own knowledge; Qwen 2.5 3B finished 0 of 7, running out of time on every task.
Figure 3. Four models on identical code (round 3, 90 minutes per task): tasks finished, and gaps filled from the model's own knowledge.

Round 3, 90 minutes per task:

Model Finished Model time Said not found Caveats Filled from own knowledge Numbers not in findings Commands not in findings Uncited lines
Qwen2.5-7B 6 of 7 197.4 min 2 2 0 0 0 10
Qwen2.5-14B 2 of 7 534.9 min 0 0 0 0 0 0
Llama-3.1-8B 2 of 7 278.0 min 2 0 0 0 0 2
Qwen2.5-3B 0 of 7 302.2 min — — — — — —

Honesty measures are summed over finished results, rescored with the 1.63 scorer.

Claim 1. Across the ten finished results from four models on identical code, none filled a gap from the model's own knowledge, and none stated a number or command absent from its findings, by the suite's measures.

Failures the smaller models exposed (each fixed in the shared code before round 3; full list in the changelog):

5.3 Untimed control (Qwen2.5-7B)

With a 180-minute cap, the 7B finished 7 of 7 in 276.5 minutes. Its results said 2 items were not found, carried 1 caveat, and had 8 uncited lines, with 0 gaps filled, 0 numbers and 0 commands not in findings. The time limit had cost only the documentation task. Without it, the remaining problems were in what got recorded and written, not in browsing:

Three fresh check runs of those three tasks followed, after the fixes in Section 5.4:

Check run Result Model time
1 Stopped after the explanation: a false "not found" (led to the partial-answer wording) —
2 3 of 3 finished 118.2 min
3 3 of 3 finished 125.6 min

5.4 Case studies

Case 1: Word overlap rated an invented command above real ones. In the untimed documentation run, the 7B recorded: "To install packages into the virtual environment, use pip install package_name inside the activated venv folder", followed by two commands it really copied. The invented sentence scored 60% word-pair overlap with the page, higher than the copied pip freeze > requirements.txt (50%) and pip install -r requirements.txt (42%). Its words echoed the objective ("install packages into it") (Figure 4).

The made-up command pip install package_name scored 60% on a word-matching check, above the two commands the AI really copied, pip freeze > requirements.txt at 50% and pip install -r requirements.txt at 42%. A new exact-match rule caught 21 of 525 past findings, every one a real problem.
Figure 4. Word-pair overlap with the page: the invented command (60%) scored above the two commands really copied (50%, 42%). The exact-text rule caught 21 of 525 saved findings.

No overlap threshold can separate these, so commands got their own rule: exact text (Section 3.2). Applied to every saved run, the rule flags 21 of 525 findings and 5 of 119 final results, all on the documentation task, and each one real when read by hand. pip install package_name alone was invented in five documentation runs, including a 14B result that presented it as found on the official page.

Case 2: Copying a table faithfully, but swapped. Copying the official venv activation table, the 7B swapped bin and Scripts (C:\> <venv>\bin\activate.bat) in two runs. Every word appears on the page, so word overlap passes it. The exact-text check catches it, because that line does not exist on the page.

Case 3: The wording of a refusal decides what the model does next. Replaying the moment of the invented command:

Refusal wording Outcome (6 samples)
"The page never gives …: do not write it." 6 of 6 kept the invented command and dropped the real ones
The same, plus the page's own line with that command 3 of 6 recorded only the page's commands; 3 of 6 repeated the refused finding, which is refused again

The second wording became the refusal.

Case 4: The wording of a yes-or-no question decides a false "not found". The coverage question asked whether a finding "says what the ITEM asks for". For a finding that answered part of an item (the light-dependent reactions, "which take place in the thylakoids"), the 7B answered no 6 times out of 6. Asked whether the finding "answers the ITEM, or part of it", it answered yes 6 of 6. The negative control, "handling of many simultaneous users", which no finding answers, stayed "no" 6 of 6 under the new wording.

When replayed faithfully, with the coverage call first as in a real run, one item still got "no" 3 of 3. The item's own distinctive words ("light", "dependent", "reaction") found in the cited finding were therefore restored as a second chance. With them, the item counted 3 of 3, and the negative control stayed unanswered 3 of 3.

Case 5: Fewer, verified send-backs. Earlier replays shaped the send-back rules:

5.5 Where our measures failed

We report these as results, because a measure that can be fooled is part of what a reader needs to know.

  1. An invented command scored as clean. The untimed documentation result was counted clean by every measure, even though it rested on pip install package_name. The scorer compares results with findings, not findings with pages. The commands column (Section 4.2) and the exact-text rule (Section 3.2) were added after this.

  2. An uncited fill. In check run 2, the explanation stated that the light-dependent reactions take place in the thylakoid membrane and the Calvin cycle in the stroma. Its only finding said the stages occur "in different locations". No measure flagged it, because the sentence shares a paragraph with cited ones. This is a gap filled from the model's knowledge in a finished 1.63 result. Claim 1 is scoped to the four-model round, but this run belongs in the record next to it.

  3. Silent omissions. Reading the results by hand, we found four finished answers that left an item out without saying "not found":

    • Windows activation, in check runs 2 and 3 of the documentation task;
    • oxygen, from the outputs of the photosynthesis explanation in check run 3;
    • a third culture, in the untimed opinion result (the cause was fixed after that run).

    The suite does not measure silent omission at all, so we cannot give a rate.

6. Negative results and dead ends

7. Limitations and threats to validity

8. Reproducibility

The code is MIT-licensed. Version 1.63 is a tagged release and is on PyPI as localagentbrowser. To run the suite on any model LM Studio serves:

python tools/regression.py --model qwen2.5-7b-instruct --minutes 90

Each run writes a folder with every task's report, journal, console output and prompt trace, plus a summary with the measures in Section 4.2. --rescore scores an earlier run again with the current scorer, as we did for Section 5.2. The offline tests (python -m pytest) run in seconds with no model or browser. The step-by-step history is in docs/regression-history.md. Code: https://github.com/SnowFXXX/LocalAgentBrowser. Release and saved runs: https://github.com/SnowFXXX/LocalAgentBrowser/releases/tag/v1.63. Project site: https://snowfxxx.github.io/LocalAgentBrowser/.

We invite replication, particularly on other models, other hardware and harder objectives. Results that contradict ours are especially welcome.

9. Conclusion and next steps

Verification by code, rather than trust in the model, produced finished results that stayed within their sources across four models from 3B to 14B parameters on a CPU, by the measures we have. The same discipline exposed where those measures fail. Those failures are now the work.

Next focus: silent omission. The fixed checklist makes it checkable. Every item must appear in the final result or be listed as not found. The difficulty is matching without false alarms, since the same thing is said in many ways (a page said "POSIX" for macOS and Linux). It is a narrow, well-defined problem and a good place for contributors. The public issue is at https://github.com/SnowFXXX/LocalAgentBrowser/issues/11.

After that:

Acknowledgements and disclosure

LocalAgentBrowser versions 1.13 onwards, and this paper, were developed with Claude (Anthropic), working in Claude Code on the Claude Pro plan with Opus 5.5 at medium effort. This was a deliberate choice. The agent's checks trust no model's word, including the one that helped build it. All numbers in this paper come from saved run folders and were recomputed from them for this draft.

References