Abstract
Language models used as research assistants produce fluent answers whose claims are often not supported by any source. They are also unreliable judges of their own work. We describe LocalAgentBrowser, an open-source autonomous web research agent built for a small model (Qwen2.5-7B-Instruct, 4-bit, CPU only, 8,192-token context), in which honesty is enforced by code rather than requested from the model. The model chooses one of five actions per turn. Every finding it records must be copied from the page shown, and deterministic checks verify that before the finding counts. Commands and paths must match the page exactly. A checklist of the objective's parts is fixed at the start. The agent is sent back only for gaps the code can verify. The final answer is written in a separate call from the recorded findings alone, and the code appends what was not found.
We evaluate on a fixed suite of seven task types run end to end on the live web. On identical code, four models (Qwen2.5 3B, 7B and 14B; Llama 3.1 8B) produced 10 finished results, none of which filled a gap from the model's own knowledge or stated a number or command absent from its findings, by the suite's lexical measures. The 7B finished 6 of 7 tasks under a 90-minute limit and 7 of 7 without one. We report the measures' blind spots as findings in their own right: an invented command the scorer counted as clean, an uncited fill, and answers that silently omitted part of the objective. Each run is saved, and the suite can be rerun.
1. Introduction
Ask a language model a research question and it answers in the voice of someone who has checked. Often nobody has. The model may have filled a gap from its training data, mixed sources, or invented a plausible detail. When challenged, many assistants apologise and promise to do better, without any mechanism that makes the next answer more reliable.
A common response is to ask the model to check itself: to cite, reflect, verify or critique. That still relies on the model's own judgement, which is the thing in doubt. This paper takes the opposite position. The model proposes; code decides what counts. We do not try to make a small model wiser. We make it hard for any model's claims to reach the final answer unless code can tie them to a page it actually read.
We built this for the hardest practical setting we could run: a 7-billion-parameter model on an ordinary office PC (Intel i5-9500, six cores, no GPU), where each model call takes from under a minute to several minutes. Small models on a CPU drift, loop, hallucinate and get stuck, and everything they do costs time. This makes their failures easy to see and expensive to ignore, which makes them a good stress test for verification by code.
Contributions.
- A design for an autonomous web research agent in which honesty properties are enforced in code: findings grounded in the page shown, with exact-text checks for commands and paths; a fixed objective checklist; code-verified coverage before finishing; and a final result written only from recorded findings, with gaps and caveats added by the code (Section 3).
- A reproducible, honesty-focused regression suite of seven task types, with measures of what finished results claim beyond their findings (Section 4).
- A cross-model evaluation on identical code (four models), plus an untimed control run (Section 5).
- Negative results and measured dead ends, including where our own measures failed (Sections 5.5 and 6).
2. Related work
Most work on making language-model agents reliable improves the model: it trains, fine-tunes or prompts the model to cite, to reflect, or to act better. We take the other route, verification instead of training. We change nothing in the model, and we move the checks out of the model and into code. We organise the related work around that difference.
Attribution and quoting. Retrieval-augmented generation (Lewis et al., 2020) conditions generation on retrieved text. Work on attribution asks whether generated statements are supported by a source: the AIS framework (Rashkin et al., 2021) and citation benchmarks such as ALCE (Gao et al., 2023). These measure attribution after the fact; our agent enforces it when a finding is recorded. The closest precedent for our core rule is GopherCite (Menick et al., 2022), which trained a 280B model to support its answers with verbatim quotes, judged by human raters. Our agent requires the same kind of quote from a 7B model without training, and code, not people, checks it against the page. Verbatim RAG, an open-source project, likewise builds answers only from extracted spans, for document question answering.
Hallucination, factuality and abstention. Surveys of hallucination (Ji et al., 2023) and fine-grained factuality scoring such as FActScore (Min et al., 2023) and SelfCheckGPT (Manakul et al., 2023) supply the measures this field uses; MiniCheck (Tang et al., 2024) shows that a 770M checker can judge support against a document about as well as GPT-4, a natural upgrade for our lexical measures (Section 5.5). The agent's "not found" behaviour is a form of abstention (Wen et al., 2025). ToolBeHonest (Zhang et al., 2024) found that tool-using models often fail to recognise unsolvable tasks and that larger models were not reliably better, which agrees with our finding that honesty did not scale with size (Section 5.2).
Self-correction. Self-Refine (Madaan et al., 2023), Reflexion (Shinn et al., 2023) and Chain-of-Verification (Dhuliawala et al., 2023) let a model revise its own output, and Self-RAG (Asai et al., 2023) trains 7B and 13B models to judge whether their own output is supported. Huang et al. (2024) found that models often cannot correct their reasoning without external feedback. Our measurements agree for a small model: about half of the 7B's own "this item is not covered" verdicts were wrong (Section 3.4). We therefore remove self-grading rather than train it.
Web agents. WebGPT (Nakano et al., 2021) trained a model to browse and cite. ReAct (Yao et al., 2023) interleaves reasoning and actions. Benchmarks include WebArena (Zhou et al., 2024) and Mind2Web (Deng et al., 2023). Open-source research tools such as GPT Researcher and Local Deep Research build multi-step, cited reports, and browser agents such as browser-use mainly target large hosted models. AgentBench (Liu et al., 2024) found a wide gap between commercial models and open models of 70B parameters or fewer.
Small and local models as agents. Most work on small agents closes that gap by changing the weights. AgentTuning (Zeng et al., 2023), FireAct (Chen et al., 2023), Lumos (Yin et al., 2023) and α-UMi (Shen et al., 2024) fine-tune open models on agent trajectories, the last splitting planning, calling and summarising across several small models. WebRL (Qi et al., 2025) uses online reinforcement learning to lift Llama-3.1-8B from 4.8% to 42.4% on WebArena-Lite. Search-R1 (Jin et al., 2025) trains Qwen2.5-3B and 7B, the family we use, to call a search engine while reasoning, and DeepResearcher (Zheng et al., 2025) trains research agents with reinforcement learning on the live web. On-device work such as TinyAgent (Erdogan et al., 2024) and Octopus v2 (Chen and Li, 2024) fine-tunes 1–7B models for fast local function calling. ACBench (Dong et al., 2025) measures what quantisation and pruning cost an agent; our model runs at 4 bits. Belcak et al. (2025) argue that small models should do most agent work, chiefly on cost. Our approach is orthogonal to all of these. We change nothing in the model, and one code path serves every model we tested (Section 3.6), so trying a new model needs no code change. Training can make an agent more capable; it cannot make the agent's claims checkable. The two compose: a WebRL- or Search-R1-trained model could run under our checks, gaining capability from training and verifiability from code.
We are not aware of a published evaluation of a 7B-class model doing open-web research on a CPU-only computer, scored on whether its answers fill gaps from the model's own knowledge. The trained research agents use GPUs and report accuracy; the on-device work concerns tool calling; GopherCite's quotes were rated by people. Our system differs from most of this work in where the check sits: in deterministic code, at the moment a claim is recorded and again when the answer is written, rather than in a model's judgement after the fact. The model never grades its own work.
3. System design
The agent is about 2,500 lines of Python in ten modules, with 251 offline tests that replace the model and the browser with scripted fakes. LM Studio serves the model through an OpenAI-compatible API, and Playwright drives a visible Chromium window.
3.1 Loop and actions
Each turn, the model sees the objective, its numbered findings, a compact journal of recent actions, and the current page part. It replies with one JSON action, constrained by a JSON schema:
| Action | Effect |
|---|---|
SEARCH |
DuckDuckGo (HTML) results with their real URLs |
NAVIGATE |
Open a URL from the results or the current page |
READ_MORE |
Next part of a long page (parts of about 2,500 characters), or the unread part whose outline label names a missing item |
DISCOVER |
Record a finding from the current page |
FINISH |
End the run, after the checks in Section 3.4 |
The agent only reads. It does not click, fill in forms or sign in.
A browsing reply is capped at 600 tokens. A 3B model once wrote more than 2,500 tokens in one reply, at about 4 tokens per second. On a CPU, reading the prompt takes about 90% of each call (about 8 prompt tokens per second on our machine). So the agent keeps one growing conversation, which lets LM Studio reuse its prompt cache. It starts a fresh prompt only near 75% of the context budget.
A refused action is corrected inside the same turn ("cheap retry"): a short note is appended, explaining why and what to do instead. This is how every check below talks to the model.
3.2 Grounding: a finding must be on the page shown
A DISCOVER is checked against the text of the page parts shown so far, in order:
- Word-pair overlap. If fewer than 70% of the finding's word pairs appear in the page text, or any sentence falls below 35%, the model is asked once to confirm, using only the page's words. Below 40% overall, or 25% for any sentence, the finding is refused outright. Short findings and sentences (fewer than 20 or 8 pairs) are not scored, because a few connecting words would sink their score.
- Pointing the way. A refusal quotes the one page sentence closest to what the model wrote, or names what the parts shown never mention ("Japan, Navajo"), so that the model can record the real text or read on. Sentences in which a page describes itself, and questions, are not quoted.
- Exact text for code. Commands and paths must appear on the page as written. This covers text in backticks,
every line of a fenced code block, and any token with a backslash, a
<placeholder>or two or more slashes; URLs and prose such as "macOS/Linux" are excluded. Case, whitespace, quote style and placeholder brackets are normalised. Section 5.4 explains why word overlap cannot catch a wrong command. - Recorded from the wrong place. A finding is refused, with the next step, if it is made only of search-result snippets, the page's headings, the page describing itself ("this guide covers…"), questions, an echo of the objective or of the model's own searches (50% overlap with the objective and less with the page), or the checklist's own wording. Each rule was added after a model did exactly that. Where possible, a rule was checked against the saved findings before it was adopted; the hits found there were junk.
3.3 A fixed objective checklist
At the start, one short call splits the objective into at most eight checklist items, keeping conditions such as "free, open-source" or "three". The checklist is fixed for the run. Every finding names the items it serves, and every later check goes through these items.
Before version 1.53, the model re-split the objective from memory at each check. It drifted: it once blocked its own finish over "migration strategies", which the objective never asked for. Fixing the checklist was the largest single improvement in the project's history (Section 5.1).
3.4 Coverage: verified in code, not taken on trust
When the model chooses FINISH, each checklist item is checked in a short call of its own. The model gives a verdict
and the evidence: the numbers of the findings, or words copied from them. The code then verifies the verdict:
- An item counts as covered only if its evidence is found in the findings. When the model gives only finding numbers twice, a yes-or-no question about that one finding is asked, and a "yes" counts only with a quote found in it.
- Items that ask for a number of things ("three suites") are counted, and things count only if a finding names them.
- Things that must meet a condition ("open-source") must be shown to meet it, in words copied from a finding about them.
Only gaps the code can verify send the agent back: an item nothing was copied for, fewer things than asked for, or nothing recorded at all. The model's other verdicts only shape the final result. This rule comes from measurement. Over the first four suite runs with a coverage check (versions 1.54 to 1.56), about half of the model's send-backs were for gaps that were not real. Examples include "missing official documentation" on a task that did not ask for it, and "H2O, O2" already recorded. Each wrong send-back cost a finish or 8 to 10 minutes.
3.5 Writing the final result
After FINISH, one more call writes the result from only the objective, the checklist and the recorded findings,
with short rules: cite findings by number, add nothing from outside, and give as many items as asked for. The code
then does what the model cannot be trusted to do:
- Not found: items no finding answers are left out of what the model is asked to write. The code appends them as "Not found in the pages read".
- Caveats: a thing not shown to meet a condition is marked where it is first named ("not shown in the pages read to be open-source").
- Filled gaps cut: a sentence of the form "not specified …, but
" is cut back to the gap. - No supplied amounts: items whose findings contain no amounts are named to the writer, so that none are supplied.
- Invented commands cut: a command no finding gives is removed (its sentence, or its line in a code block) and named after the result ("Left out, because no discovery gives them").
- Numbers flagged: any number in the result that appears in no finding is flagged in the report.
The report always lists every finding with its source URL, however the run ends.
3.6 One code path for every model
Every rule applies to every model. There are no per-model prompts, branches or profiles. When a new model exposed a failure (Section 5.2), the fix went into the shared code and was measured on the default model too.
4. Evaluation method
4.1 The suite
Seven objectives, one of each task type:
| Task | Objective (abridged) | What it tests |
|---|---|---|
| Factual | Tallest mountain in Africa, height in meters and feet | Basic retrieval and finishing |
| List | Three free, open-source alternatives to Microsoft Office, and what each is best at | Counts and conditions |
| Comparison | SQLite vs PostgreSQL for a small web app, then a recommendation | Multiple items and reporting a gap |
| How-to | Cold brew coffee: equipment, steps with amounts, how long it keeps | Amounts and step completeness |
| Explanation | How photosynthesis works: inputs, outputs, both stages and where | Multi-part coverage on long pages |
| Opinion | How at least three cultures have viewed owls, then an opinion | Counting, and an opinion kept separate |
| Documentation | From the official Python docs: create, activate (Windows; macOS/Linux) and install into a venv | Source conditions and exact commands |
4.2 Measures
For each task: finished or not, turns, model minutes, and how many checklist items have findings. For each finished result, the suite counts:
- Honest gaps: items it says were not found, and code caveats.
- Claims beyond the findings:
- gaps filled from the model's own knowledge, detected lexically ("the text does not specify, but…");
- numbers no finding contains;
- commands no finding contains (added in 1.63);
- uncited lines;
- citations of findings that do not exist.
All of these measures are lexical. They detect certain patterns, and they cannot judge meaning. Section 5.5 gives their known misses.
4.3 Setup
- Hardware: Intel i5-9500 (6 cores), no GPU.
- Server: LM Studio, with Q4_K_M quantisation for every model and an 8,192-token context.
- Time limits:
- Version history: 45 minutes per task, 60 for the explanation.
- Cross-model round: 90 minutes per task for every model.
- Untimed control: a 180-minute cap that no task approached.
4.4 Process rules
- Every run starts fresh and runs from start to finish.
- No code changes during a run.
- If a run exposed a failure with a known fix, it was stopped, the fix was made for every model, and that model's suite was restarted from scratch ("stop and fix").
- A run in which the time limit hit while the final result was being written is not counted as finished. Its unchecked draft is kept but not scored. (This rule was added during this evaluation; see Section 5.2.)
4.5 Replays
Many decisions were tested on a single saved moment instead of a whole suite. The exact prompt from a run was sent to the model 3 to 9 times, before and after a change. This tests a change on the moment it is meant for, in minutes rather than hours.
4.6 Scale
From 2026-10-16 to 2026-10-29 we ran 44 suite runs on one machine. They comprised 205 task runs, recorded 525 findings and took 118.7 hours of model time. Every run folder keeps the prompts, the model's replies, the journal and the report (Figure 1).
5. Results
5.1 Progress over versions (Qwen2.5-7B)
| Version | 1.52 | 1.53 | 1.54 | 1.54b | 1.55 | 1.56 | 1.57 | 1.58 | 1.59 | 1.60 | 1.61 | 1.62 | 1.63 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Finished (of 7) | 2 | 5 | 4 | 5 | 5 | 4 | 4 | 5 | 5 | 6 | 6 | 7 | 6 |
| Model minutes | 254 | 200 | 239 | 225 | 233 | 272 | 270 | 214 | 209.5 | 170.6 | 222.1 | 168.3 | 197.4 |
Version 1.62 changed no behaviour (a code reorganisation verified by golden-prompt tests), so its 7 of 7 against 1.61's 6 of 7 is run-to-run variance. One run per version cannot separate small effects from variance, and we do not claim it does.
The large steps were the fixed checklist (1.53: 2 → 5) and sending the agent back only for gaps the code verifies (1.58). Several versions finished fewer tasks because they were stricter: a check that refuses more costs turns, and each turn costs minutes.
5.2 Four models, identical code
Round 3, 90 minutes per task:
| Model | Finished | Model time | Said not found | Caveats | Filled from own knowledge | Numbers not in findings | Commands not in findings | Uncited lines |
|---|---|---|---|---|---|---|---|---|
| Qwen2.5-7B | 6 of 7 | 197.4 min | 2 | 2 | 0 | 0 | 0 | 10 |
| Qwen2.5-14B | 2 of 7 | 534.9 min | 0 | 0 | 0 | 0 | 0 | 0 |
| Llama-3.1-8B | 2 of 7 | 278.0 min | 2 | 0 | 0 | 0 | 0 | 2 |
| Qwen2.5-3B | 0 of 7 | 302.2 min | — | — | — | — | — | — |
Honesty measures are summed over finished results, rescored with the 1.63 scorer.
- 7B: only the documentation task ran out of time, with all four items found.
- 14B: about 2.5 times the 7B's time per call (calls of 400–600 s). It reached
FINISHon five tasks, but on three of them the time limit hit while the final result was being checked and written. The run's original summary counted those unchecked drafts as finished (5 of 7). One of them contained an invented command. The harness now marks such runs "stopped while writing", and the table counts only the two clean finishes. - Llama 3.1 8B: browsing was mostly sound, but its pacing was slow (it read on after having the answer), and five tasks timed out with most items found.
- Qwen2.5 3B: recorded findings on every task (3 of 3 items on the factual task, 4 of 4 on the documentation task) but never finished one. Late in the round it also recorded general, weakly grounded lines. We treat 0 of 7 as this model's floor at this setting and will test another 3B model to separate size from model family.
Claim 1. Across the ten finished results from four models on identical code, none filled a gap from the model's own knowledge, and none stated a number or command absent from its findings, by the suite's measures.
Failures the smaller models exposed (each fixed in the shared code before round 3; full list in the changelog):
- Runaway replies: a 3B reply of 2,500+ tokens.
- Search-result snippets recorded as findings: the 3B did this in 5 of 7 tasks, and every model did it at least once.
- A made-up tool call instead of a final result: asked for plain text, Llama 3.1 8B answered 3 of 6 final results with a JSON tool call. Asking for the result in a required JSON field fixed it.
- Echoes of the objective recorded as findings.
- A model call longer than the client's 10-minute default: the 14B. The timeout is now 30 minutes.
5.3 Untimed control (Qwen2.5-7B)
With a 180-minute cap, the 7B finished 7 of 7 in 276.5 minutes. Its results said 2 items were not found, carried 1 caveat, and had 8 uncited lines, with 0 gaps filled, 0 numbers and 0 commands not in findings. The time limit had cost only the documentation task. Without it, the remaining problems were in what got recorded and written, not in browsing:
- The explanation listed its outputs as "not found" over a finding that gave them (a false "not found").
- The opinion left out a culture its findings named (a silent omission).
- The documentation task recorded
pip install package_name, which neither page it read gives (an invented command in a finding).
Three fresh check runs of those three tasks followed, after the fixes in Section 5.4:
| Check run | Result | Model time |
|---|---|---|
| 1 | Stopped after the explanation: a false "not found" (led to the partial-answer wording) | — |
| 2 | 3 of 3 finished | 118.2 min |
| 3 | 3 of 3 finished | 125.6 min |
5.4 Case studies
Case 1: Word overlap rated an invented command above real ones. In the untimed documentation run, the 7B recorded:
"To install packages into the virtual environment, use pip install package_name inside the activated venv folder",
followed by two commands it really copied. The invented sentence scored 60% word-pair overlap with the page, higher
than the copied pip freeze > requirements.txt (50%) and pip install -r requirements.txt (42%). Its words echoed
the objective ("install packages into it") (Figure 4).
No overlap threshold can separate these, so commands got their own rule: exact text (Section 3.2). Applied to every
saved run, the rule flags 21 of 525 findings and 5 of 119 final results, all on the documentation task, and each one
real when read by hand. pip install package_name alone was invented in five documentation runs, including a 14B result
that presented it as found on the official page.
Case 2: Copying a table faithfully, but swapped. Copying the official venv activation table, the 7B swapped bin
and Scripts (C:\> <venv>\bin\activate.bat) in two runs. Every word appears on the page, so word overlap passes it.
The exact-text check catches it, because that line does not exist on the page.
Case 3: The wording of a refusal decides what the model does next. Replaying the moment of the invented command:
| Refusal wording | Outcome (6 samples) |
|---|---|
"The page never gives …: do not write it." |
6 of 6 kept the invented command and dropped the real ones |
| The same, plus the page's own line with that command | 3 of 6 recorded only the page's commands; 3 of 6 repeated the refused finding, which is refused again |
The second wording became the refusal.
Case 4: The wording of a yes-or-no question decides a false "not found". The coverage question asked whether a finding "says what the ITEM asks for". For a finding that answered part of an item (the light-dependent reactions, "which take place in the thylakoids"), the 7B answered no 6 times out of 6. Asked whether the finding "answers the ITEM, or part of it", it answered yes 6 of 6. The negative control, "handling of many simultaneous users", which no finding answers, stayed "no" 6 of 6 under the new wording.
When replayed faithfully, with the coverage call first as in a real run, one item still got "no" 3 of 3. The item's own distinctive words ("light", "dependent", "reaction") found in the cited finding were therefore restored as a second chance. With them, the item counted 3 of 3, and the negative control stayed unanswered 3 of 3.
Case 5: Fewer, verified send-backs. Earlier replays shaped the send-back rules:
- A count rule (1.60): the 1.59 owl finish with two tribes for three cultures was sent back 2 of 2 times, while the earlier owl moments with three or four cultures were sent back 0 of 6.
- Finish advice in the read-on check: read on 6 of 6 before, chose FINISH 9 of 9 after.
- The outline pointer (1.61): told only to use the page outline, the model jumped to the right part 0 of 6
times; told which part, 3–4 of 6; with the code making the jump on a plain
READ_MORE, 9 of 9.
5.5 Where our measures failed
We report these as results, because a measure that can be fooled is part of what a reader needs to know.
-
An invented command scored as clean. The untimed documentation result was counted clean by every measure, even though it rested on
pip install package_name. The scorer compares results with findings, not findings with pages. The commands column (Section 4.2) and the exact-text rule (Section 3.2) were added after this. -
An uncited fill. In check run 2, the explanation stated that the light-dependent reactions take place in the thylakoid membrane and the Calvin cycle in the stroma. Its only finding said the stages occur "in different locations". No measure flagged it, because the sentence shares a paragraph with cited ones. This is a gap filled from the model's knowledge in a finished 1.63 result. Claim 1 is scoped to the four-model round, but this run belongs in the record next to it.
-
Silent omissions. Reading the results by hand, we found four finished answers that left an item out without saying "not found":
- Windows activation, in check runs 2 and 3 of the documentation task;
- oxygen, from the outputs of the photosynthesis explanation in check run 3;
- a third culture, in the untimed opinion result (the cause was fixed after that run).
The suite does not measure silent omission at all, so we cannot give a rate.
6. Negative results and dead ends
- Quoting a page sentence in the read-on check. To stop the 7B reading all twelve parts of the venv reference page without recording anything, we tried quoting the page sentence that best matched a missing item. Neither our closest-sentence measure nor a rarity-weighted variant could separate good picks ("activated using a script in its binary directory") from API-reference junk ("Installs activation scripts…"). They scored the same. Because the 7B follows quoted hints, this would have produced junk findings. We removed it.
- Showing the writer the gaps the check could not confirm. Listing "possibly missing" items to the final-result call did not stop gap filling. One sample filled them in anyway, and another produced empty headings. We removed it.
- Linking findings to items by shared words. In the saved journals, 75 of 247 finding-to-item links share no key word with their item, and most of those are right ("5,895 m" for "how tall in meters"). We did not adopt it.
- A larger context. 16,384 tokens instead of 8,192 would roughly halve the fresh prompts that cost 4–5 minutes each, but it needs more memory. We kept the default where a modest PC can run it.
7. Limitations and threats to validity
- Sample size. One run per model and per version, on seven tasks. Our claims are counts of what happened in recorded runs, not rates with confidence intervals. Run-to-run variance is large (Section 5.1).
- One machine, one quantisation. Q4_K_M, CPU only, a fixed context. Time limits interact with speed: the 14B's score is mostly a speed result.
- The live web. Pages change, and a single site can decide a task. The suite flags a task that fails again on the same site.
- Lexical measures. Section 5.5. A result can be counted clean and still be wrong.
- Open problems:
- text copied faithfully but under the wrong label (a Git Bash line labelled as the macOS/Linux command);
- silent omission;
- the final result filling gaps the coverage check could not confirm;
- small embellishments while writing (every opinion sample called Native American owls "wise", which no finding says).
- The author built the suite and the agent, and the system was developed with an AI assistant (see the disclosure). Both are reasons for outside replication, which Section 8 invites.
8. Reproducibility
The code is MIT-licensed. Version 1.63 is a tagged release and is on PyPI as localagentbrowser. To run the suite on
any model LM Studio serves:
python tools/regression.py --model qwen2.5-7b-instruct --minutes 90
Each run writes a folder with every task's report, journal, console output and prompt trace, plus a summary with the
measures in Section 4.2. --rescore scores an earlier run again with the current scorer, as we did for Section 5.2.
The offline tests (python -m pytest) run in seconds with no model or browser. The step-by-step history is in
docs/regression-history.md. Code: https://github.com/SnowFXXX/LocalAgentBrowser. Release and saved runs: https://github.com/SnowFXXX/LocalAgentBrowser/releases/tag/v1.63. Project site: https://snowfxxx.github.io/LocalAgentBrowser/.
We invite replication, particularly on other models, other hardware and harder objectives. Results that contradict ours are especially welcome.
9. Conclusion and next steps
Verification by code, rather than trust in the model, produced finished results that stayed within their sources across four models from 3B to 14B parameters on a CPU, by the measures we have. The same discipline exposed where those measures fail. Those failures are now the work.
Next focus: silent omission. The fixed checklist makes it checkable. Every item must appear in the final result or be listed as not found. The difficulty is matching without false alarms, since the same thing is said in many ways (a page said "POSIX" for macOS and Linux). It is a narrow, well-defined problem and a good place for contributors. The public issue is at https://github.com/SnowFXXX/LocalAgentBrowser/issues/11.
After that:
- a measure for silent omission in the suite itself;
- another 3B model, to separate size from model family;
- a trained agent model (for example from WebRL or Search-R1) run under the same checks, to test whether training and code-enforced verification add up;
- untimed suites for every model;
- and, eventually, clicking and forms, which the agent does not do yet.
Acknowledgements and disclosure
LocalAgentBrowser versions 1.13 onwards, and this paper, were developed with Claude (Anthropic), working in Claude Code on the Claude Pro plan with Opus 5.5 at medium effort. This was a deliberate choice. The agent's checks trust no model's word, including the one that helped build it. All numbers in this paper come from saved run folders and were recomputed from them for this draft.
References
- Asai, A., Wu, Z., Wang, Y., Sil, A. and Hajishirzi, H. (2023). Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection. arXiv:2310.11511.
- Belcak, P. et al. (2025). Small Language Models are the Future of Agentic AI. arXiv:2506.02153.
- browser-use. Open-source project. https://github.com/browser-use/browser-use
- Chen, B. et al. (2023). FireAct: Toward Language Agent Fine-tuning. arXiv:2310.05915.
- Chen, W. and Li, Z. (2024). Octopus v2: On-device language model for super agent. arXiv:2404.01744.
- Deng, X. et al. (2023). Mind2Web: Towards a Generalist Agent for the Web. NeurIPS 2023. arXiv:2306.06070.
- Dhuliawala, S. et al. (2023). Chain-of-Verification Reduces Hallucination in Large Language Models. Findings of ACL 2024. arXiv:2309.11495.
- Dong, P. et al. (2025). Can Compressed LLMs Truly Act? An Empirical Evaluation of Agentic Capabilities in LLM Compression. ICML 2025. arXiv:2505.19433.
- Erdogan, L. E. et al. (2024). TinyAgent: Function Calling at the Edge. EMNLP 2024 (demo). arXiv:2409.00608.
- Gao, T., Yen, H., Yu, J. and Chen, D. (2023). Enabling Large Language Models to Generate Text with Citations. EMNLP 2023. arXiv:2305.14627.
- GPT Researcher. Open-source project. https://github.com/assafelovic/gpt-researcher
- Huang, J. et al. (2024). Large Language Models Cannot Self-Correct Reasoning Yet. ICLR 2024. arXiv:2310.01798.
- Ji, Z. et al. (2023). Survey of Hallucination in Natural Language Generation. ACM Computing Surveys. arXiv:2202.03629.
- Jin, B. et al. (2025). Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning. arXiv:2503.09516.
- Lewis, P. et al. (2020). Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. NeurIPS 2020. arXiv:2005.11401.
- Liu, X. et al. (2024). AgentBench: Evaluating LLMs as Agents. ICLR 2024. arXiv:2308.03688.
- Local Deep Research. Open-source project. https://github.com/LearningCircuit/local-deep-research
- Madaan, A. et al. (2023). Self-Refine: Iterative Refinement with Self-Feedback. NeurIPS 2023. arXiv:2303.17651.
- Manakul, P., Liusie, A. and Gales, M. J. F. (2023). SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models. EMNLP 2023. arXiv:2303.08896.
- Menick, J. et al. (2022). Teaching language models to support answers with verified quotes. arXiv:2203.11147.
- Min, S. et al. (2023). FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation. EMNLP 2023. arXiv:2305.14251.
- Nakano, R. et al. (2021). WebGPT: Browser-assisted question-answering with human feedback. arXiv:2112.09332.
- Qi, Z. et al. (2025). WebRL: Training LLM Web Agents via Self-Evolving Online Curriculum Reinforcement Learning. ICLR 2025. arXiv:2411.02337.
- Rashkin, H. et al. (2021). Measuring Attribution in Natural Language Generation Models. arXiv:2112.12870.
- Shen, W. et al. (2024). Small LLMs Are Weak Tool Learners: A Multi-LLM Agent. EMNLP 2024. arXiv:2401.07324.
- Shinn, N. et al. (2023). Reflexion: Language Agents with Verbal Reinforcement Learning. NeurIPS 2023. arXiv:2303.11366.
- Tang, L., Laban, P. and Durrett, G. (2024). MiniCheck: Efficient Fact-Checking of LLMs on Grounding Documents. EMNLP 2024. arXiv:2404.10774.
- Verbatim RAG. Open-source project. https://pypi.org/project/verbatim-rag/
- Wen, B. et al. (2025). Know Your Limits: A Survey of Abstention in Large Language Models. TACL. arXiv:2407.18418.
- Yao, S. et al. (2023). ReAct: Synergizing Reasoning and Acting in Language Models. ICLR 2023. arXiv:2210.03629.
- Yin, D. et al. (2023). Lumos: Learning Agents with Unified Data, Modular Design, and Open-Source LLMs. arXiv:2311.05657.
- Zeng, A. et al. (2023). AgentTuning: Enabling Generalized Agent Abilities for LLMs. Findings of ACL 2024. arXiv:2310.12823.
- Zhang, Y. et al. (2024). ToolBeHonest: A Multi-level Hallucination Diagnostic Benchmark for Tool-Augmented Large Language Models. EMNLP 2024. arXiv:2406.20015.
- Zheng, Y. et al. (2025). DeepResearcher: Scaling Deep Research via Reinforcement Learning in Real-world Environments. EMNLP 2025. arXiv:2504.03160.
- Zhou, S. et al. (2024). WebArena: A Realistic Web Environment for Building Autonomous Agents. ICLR 2024. arXiv:2307.13854.