Every prompt my AIOS sends went to the big cloud model, out of habit. So I tested 12 free local models on a real job, wrote every score to a database, and built a router that reads it. It names the cheapest model that measured good enough for that kind of task. And when it has no data, it says so out loud instead of sounding sure.
My AIOS sends prompts all day: inbox rules, thread summaries, small scripts, Swedish replies. By default every one went to a frontier cloud model, because nobody knew which cheaper model was good enough for which job. Meanwhile a Mac mini in the corner serves twenty-odd free models through Ollama and mostly sits idle. The question was never “are local models good?” It was “good at what, measured how?” So the goal I wrote down on 2026-10-08: every prompt goes to the cheapest model known to do that kind of task well, Claude as the ceiling, the mini as the private floor, and a database of measured results deciding instead of a hunch.
The test was a job Claude had just done for me: the scripts that clean up my Gmail. I cut it into five function-level tasks (aggregate senders, classify a sender, build Gmail filters, plan a backfill, apply and undo), gave every model the same spec, and let hidden tests grade the code. Wall time recorded, sandboxed, run on the mini. No real mail went anywhere near a model: make_fixture.py invents every sender, “same record shape, no real people”. The result: five models pass nearly everything. qwen3-coder:30b scores 4.8 of 5 in 50 seconds for all five tasks; muse-glimmer:30b gets a perfect 5.0 but needs 2,090 seconds. Two models score zero across the board. Every one of those numbers went into a SQLite database that now holds 23 models.
qwen3-coder:30b 4.8 50smuse-glimmer:30b 5.0 2090sgemma4:26b 0.0 823s
route.py decide "<prompt>" does three things. It classifies the task kind with rules (the caller’s hint, eleven regexes, a Swedish-word share). It asks the database one question, db.py recommend. And it applies three fixed rules that sit above the data: privacy forces local, high stakes force Claude, and no data, no local: a kind with fewer than three measured runs per model goes to Claude and gets flagged as an eval gap. Between those rules, the numbers decide: 0.9 or better routes local; 0.6 to 0.9 routes local only if the caller can verify the answer; and among the models that qualify, the fastest wins, not the highest score. It is a dry run for now: it prints the decision and the exit code says what kind (0 = pick, 2 = degrade, 3 = kind unknown).
kind: code-function (rule:code-request)decision: qwen3-coder:30b [local-verified]
Only code-function has three runs per model. Everything else in the database has one or two, so rule 3 blocks it. Ask the router to write a short Swedish reply with --privacy and it answers decision: degrade-or-chunk, no candidate, eval gap: 21 models have fewer than 3 runs, exit 2. With --min-runs 2 it would happily pick three models at 1.00 in under six seconds, on two data points each. Production keeps the rule. The data also moved under me the same day: once the aios-12 prompts counted alongside Gmail, qwen3-coder:30b dropped to 0.83, so it now leads only when the caller can check its answer. That is the real state: the router has data for code and nothing else, and it says exactly that.
The eval-gap warning, the 18 tests, the rules and thresholds, and the five decisions I signed off before a line of router code. Tap any image to enlarge it and read the exact prompt that drew it.




The cheap part was never the model. It was the measurement. Once real scores sit in a table, “which model?” becomes a query, and the expensive default becomes a choice you can see. But the part I trust most is the refusal: a router that guesses from model marketing will be confidently wrong on the kinds nobody tested. Mine routes code locally because it has three runs per model saying it can, and for everything else it says “eval gap” and sends the prompt to Claude. The fix for a gap is cheap and boring: one more test prompt per kind. If you are thinking about mixing local and cloud models, start with a table of your own scores, not a leaderboard.
Twelve local models on one real job, the score table, and a router that picks a model for code and admits an eval gap for everything else.
The pattern from this episode: a small eval family with hidden tests, a SQLite table of model × task-kind scores, and a stdlib-only router that reads it under three fixed rules and flags every kind it has no data for. Everything in this episode is free and open — clone it, run it, make it yours.
db.py recommend: the one query the router makes
cmd py scripts/route.py decide "<prompt>" --text
template the three rules + threshold table
assets an eval family: tasks.json + tests.py + fixture/ (synthetic mail)
# repo soon. Python 3 stdlib + SQLite: py scripts/route.py decide "write a function that groups senders by domain" --verifiable --max-wait 60 --text
No GitHub? Comment NODATA on the post and the bot DMs you the link.
Next: route.py run. It calls the pick, checks the answer with the same hidden tests, retries once with the failure text, falls through to the cloud, and writes every attempt back to the database. Five Gmail prompts went through it. How many fell through?