Telling Technology · Learning in public
All episodes
EP.121 — NO DATA, NO GUESS
Routing in public · the router that says “no data”

I stopped guessing which AI model to use

Every prompt my AIOS sends went to the big cloud model, out of habit. So I tested 12 free local models on a real job, wrote every score to a database, and built a router that reads it. It names the cheapest model that measured good enough for that kind of task. And when it has no data, it says so out loud instead of sounding sure.

A dark switchboard titled Which model? Ask the scores: task kinds on the left patched by cables to models on the right, code to qwen3-coder and summary to the Sonnet cloud rung, a score card with each model's score and wall time, and the swedish socket left unplugged with an amber tag reading no data, eval gap
⌁ 12 local models · 5 tasks · 23 models in the DB · 18 tests · 2026-10-08
first, the expensive habit
01 The habit

Every prompt went to the most expensive model

My AIOS sends prompts all day: inbox rules, thread summaries, small scripts, Swedish replies. By default every one went to a frontier cloud model, because nobody knew which cheaper model was good enough for which job. Meanwhile a Mac mini in the corner serves twenty-odd free models through Ollama and mostly sits idle. The question was never “are local models good?” It was “good at what, measured how?” So the goal I wrote down on 2026-10-08: every prompt goes to the cheapest model known to do that kind of task well, Claude as the ceiling, the mini as the private floor, and a database of measured results deciding instead of a hunch.

claude code — desktop
> which local model should write a Python function for my Gmail cleanup?
Honest answer before this project: no idea. There were model cards and benchmark tables, but no measurement on my kind of work.
Paraphrased, not a transcript.
A diagram: many small prompt boxes on the left, every arrow going to one large cloud box labelled frontier model with a coin icon, while a Mac mini box at the bottom with twenty model chips sits greyed out and idle
before
Everything to the cloud, the mini idle.
02 The eval

12 local models, one real job, hidden tests

The test was a job Claude had just done for me: the scripts that clean up my Gmail. I cut it into five function-level tasks (aggregate senders, classify a sender, build Gmail filters, plan a backfill, apply and undo), gave every model the same spec, and let hidden tests grade the code. Wall time recorded, sandboxed, run on the mini. No real mail went anywhere near a model: make_fixture.py invents every sender, “same record shape, no real people”. The result: five models pass nearly everything. qwen3-coder:30b scores 4.8 of 5 in 50 seconds for all five tasks; muse-glimmer:30b gets a perfect 5.0 but needs 2,090 seconds. Two models score zero across the board. Every one of those numbers went into a SQLite database that now holds 23 models.

claude code — desktop
> py scripts/db.py matrix --family gmail-cleanup
qwen3-coder:30b    4.8   50s
muse-glimmer:30b   5.0 2090s
gemma4:26b         0.0  823s
12 models, 5 tasks, score 1 = all hidden tests pass. Three rows of twelve, real output.
A dark score table: twelve local models down the side, five Gmail cleanup tasks across the top, cells shaded teal for a full pass and coral for zero, with a total and a wall-time column; qwen3-coder 30b highlighted at 4.8 in 50 seconds
db.py matrix
Speed matters as much as the score.
03 The router

Ask “which model?” and get data back

route.py decide "<prompt>" does three things. It classifies the task kind with rules (the caller’s hint, eleven regexes, a Swedish-word share). It asks the database one question, db.py recommend. And it applies three fixed rules that sit above the data: privacy forces local, high stakes force Claude, and no data, no local: a kind with fewer than three measured runs per model goes to Claude and gets flagged as an eval gap. Between those rules, the numbers decide: 0.9 or better routes local; 0.6 to 0.9 routes local only if the caller can verify the answer; and among the models that qualify, the fastest wins, not the highest score. It is a dry run for now: it prints the decision and the exit code says what kind (0 = pick, 2 = degrade, 3 = kind unknown).

claude code — desktop
> py scripts/route.py decide "Write a Python function that groups Gmail senders by domain and returns counts" --verifiable --max-wait 60 --text
kind: code-function (rule:code-request)
decision: qwen3-coder:30b [local-verified]
1. qwen3-coder:30b · n=7 · 0.83 · 15s
2. mistral-small3.2:24b · 0.75 · 42s
3. gpt-oss:20b · 0.70 · 50s
4. claude-sonnet-5-5 · cloud rung
Real output, trimmed. Exit 0.
A dark terminal running route.py decide on a Gmail function prompt: kind code-function, decision qwen3-coder 30b local-verified, then a ranked list of three local models with run counts, scores and wall times, a Sonnet cloud rung, and lists of skipped models with reasons
exit 0
Three local picks, the cloud as the net.
⚠️

The honest part: it can only route one kind of task today

Only code-function has three runs per model. Everything else in the database has one or two, so rule 3 blocks it. Ask the router to write a short Swedish reply with --privacy and it answers decision: degrade-or-chunk, no candidate, eval gap: 21 models have fewer than 3 runs, exit 2. With --min-runs 2 it would happily pick three models at 1.00 in under six seconds, on two data points each. Production keeps the rule. The data also moved under me the same day: once the aios-12 prompts counted alongside Gmail, qwen3-coder:30b dropped to 0.83, so it now leads only when the caller can check its answer. That is the real state: the router has data for code and nothing else, and it says exactly that.

04 Receipts

What the router knows, and what it admits it doesn’t

The eval-gap warning, the 18 tests, the rules and thresholds, and the five decisions I signed off before a line of router code. Tap any image to enlarge it and read the exact prompt that drew it.

MODEL-ROUTER · P1 BUILT 2026-10-08

A router that admits “no data” beats one that always sounds sure

The cheap part was never the model. It was the measurement. Once real scores sit in a table, “which model?” becomes a query, and the expensive default becomes a choice you can see. But the part I trust most is the refusal: a router that guesses from model marketing will be confidently wrong on the kinds nobody tested. Mine routes code locally because it has three runs per model saying it can, and for everything else it says “eval gap” and sends the prompt to Claude. The fix for a gap is cheap and boring: one more test prompt per kind. If you are thinking about mixing local and cloud models, start with a table of your own scores, not a leaderboard.

A bold typographic card on near-black: a score table fading into an amber cell reading no data, and the line a router that admits no data beats one that always sounds sure
12 models evaluated · 5 tasks · 23 models in the DB · 3 fixed rules · 18 tests · D1–D5 logged
06 Steal this

A score table that picks your model

The pattern from this episode: a small eval family with hidden tests, a SQLite table of model × task-kind scores, and a stdlib-only router that reads it under three fixed rules and flags every kind it has no data for. Everything in this episode is free and open — clone it, run it, make it yours.

skill db.py recommend: the one query the router makes cmd py scripts/route.py decide "<prompt>" --text template the three rules + threshold table assets an eval family: tasks.json + tests.py + fixture/ (synthetic mail)
run it # repo soon. Python 3 stdlib + SQLite: py scripts/route.py decide "write a function that groups senders by domain" --verifiable --max-wait 60 --text

No GitHub? Comment NODATA on the post and the bot DMs you the link.

Next episode

My router now grades its own answers

Next: route.py run. It calls the pick, checks the answer with the same hidden tests, retries once with the failure text, falls through to the cloud, and writes every attempt back to the database. Five Gmail prompts went through it. How many fell through?

A darkened teaser image of a routing map with prompts flowing to different destinations
drops soon · follow so you don’t miss it