We Can Do Better: A Small AI That Can't Cover Up
From an Apple IIe lab in 1988 to a 7B model on an office PC: why honesty that's checked holds up better than honesty that's promised.
Earlier: Teaching a 7B model on a CPU to finish what it starts.
An AI that can't cover up. Give it a question: a model, local by default, searches the web, reads pages and copies what it finds, and code checks every copy against the page before it counts.
L.A.B. for short, in a LAB COAT: Citations Over Assumed Truth. Our principle is Verification Over Trust.
New in 1.66: sturdier cloud runs, more of every answer checked in code, the console and browser side by side, and 85% on the standard 200 SimpleQA questions with Phi-4.
The rules aren't a promise the model makes. They're written in code, and the code checks every claim. These are claims about our system, measured on a public test suite of seven kinds of question.
Four local models on 1.64 and four cloud models on 1.66, each set on the same code. No finished answer filled a gap with the model's own knowledge, by the suite's measures.
Every finding is copied from a page and checked against it. Commands must appear exactly as the page writes them.
Reading results by hand, we still find answers that leave a part out without a word, and findings that change one key word of the page ("foil" where the page says "épée") yet pass the word check. Our tests don't count either yet. That's our next focus, and you can help.
On your own computer, measured on version 1.64: the same seven tasks, the same code, 90 minutes per task, on an ordinary 6-core office PC with no graphics card. None of them filled a gap in a finished answer from its own knowledge. 1.66 changes the agent for every model and was measured on cloud models (below); these local results are 1.64's, not yet re-measured.
The one it was built with. The right start for most PCs.
No invented facts in any answer. When it errs, it errs cautious.
About 2.5 times slower on a CPU; its two time-outs came while a correct answer was being checked.
For PCs with little memory. One finished answer has a claim the checks missed.
Or a cloud model, for fast answers. The same code can use a model in the cloud through your own OpenRouter account: a few minutes instead of an hour, but it costs money, and your prompts and the pages read leave your computer. Off unless you turn it on. Measured on 1.66, one run each, one model at a time:
The cleanest results for the price. 85% on 200 SimpleQA questions.
Every expected detail in every answer, by the suite's word checks.
The fastest. Its answers used to be lost when it repeated the prompt; 1.66 keeps them.
6 of 7 on 1.65. One answer gave a glacier's elevation as Kilimanjaro's height, which no check caught.
No finished answer filled a gap from the model's own knowledge, by the suite's measures. On 1.65.1, Gemini 3.8 Flash and GPT-5.6 Terra also finished 7 of 7 (for $0.24 and $0.47), and GPT-4o lost three tasks to a rate limit that 1.66 now waits out. On SimpleQA, a public benchmark of short, hard facts, Phi-4 with LocalAgentBrowser 1.66 answered 170 of the standard 200 (85%, likely range 79 to 89%), and was right 94% of the time when it answered. 19 times it said it couldn't verify an answer rather than guess; 11 answers were wrong. Alone, with no browsing, Phi-4 answered 1 of 50; with LocalAgentBrowser, 45 of the same 50. On a CPU with no GPU, Qwen2.5-7B answered 35 of 50 (the Fathom-DeepResearch paper reports 50.8% for that model with plain search, on a different question set). Graded by Claude, judgement calls reviewed by the author, and checked against OpenAI's own simple-evals grader, which gives the same grade on 194 of the 200. Every answer is published, also as a Hugging Face dataset.
On version 1.63 the four local models finished 6, 2, 2 and 0 of 7. Every run, measured.
Each turn, the model sees the question, what it has found so far and the current page, and picks exactly one action. A visible Chromium window lets you watch every step.
SEARCHSearches DuckDuckGo and shows the results with their real addresses.
NAVIGATEOpens a page from the results or from the current page's links.
READ_MOREReads the next part of a long page, or jumps to the part whose heading names what's still missing.
DISCOVERRecords a finding, copied from the page. Code checks it is really there before it counts.
FINISHEach part of the question is checked against the findings. The answer is then written only from what was found, and anything that wasn't is listed as not found.
A finding the page doesn't support is turned away, the model is told why, and it's sent back to the page.
No graphics card needed. A run takes from about 10 minutes to an hour on a CPU, so start it and let it work. You'll need LM Studio (free), which runs the model on your computer, unless you use cloud models.
OPENROUTER_API_KEY environment variable
instead.Python 3.10 or later, in a virtual environment:
python -m venv venv
venv\Scripts\activate
pip install localagentbrowser
python -m playwright install chromium
localagentbrowser
On macOS or Linux, activate with source venv/bin/activate. Cloud models without the
installer: the settings are in the README's FAQ.
In LM Studio, load the model with a context length of 8192 and start the local server.
Local by default: searches go to DuckDuckGo and pages load like in any browser, while the model and the report stay on your computer. Cloud models are an option you turn on yourself, with your own key; then your prompts and the pages read go to the cloud provider.
See three example reports, unedited, with what each got right and wrong.
We can do better. If you think we're wrong, run it and show us.
The code, the test suite and every run are public. Pick a different model, write a harder question, find the answer it makes up, and tell us. And if you think we're right, help with the silent omissions. We named our own misses first.
# the seven-task suite: four hours or more on a CPU
python tools/regression.py --model qwen2.5-7b-instruct --minutes 90
From an Apple IIe lab in 1988 to a 7B model on an office PC: why honesty that's checked holds up better than honesty that's promised.
Earlier: Teaching a 7B model on a CPU to finish what it starts.