Telling Technology · Learning in public
All episodes
EP.126 — INVENTED NONE
A local model, a test that counts lies

My voice AI invented 7 answers out of 24. Then it invented none.

I asked the voice assistant on my Mac mini 24 questions about my own sessions, todos and projects. In 7 answers it made something up. No bigger model fixed it: a scoring script and four strict lines in the prompt did.

A phone on a dark desk with a voice assistant speech bubble; one date inside the bubble is struck out in coral and replaced by the words not in my notes
qwen3.6 on a Mac mini · 24 questions · 7 invented → 0
It sounded sure of itself. It was wrong.
01 The problem

It sounds great right up to the moment it is confidently wrong

My voice page on the Mac mini (the one from My AI started talking back) now reads a brief: a snapshot of my projects, open todos and the Claude Code sessions of the last 48 hours. So I can ask “anything coming due?” out loud and get an answer from my real data. The trouble: a small local model fills gaps with things that sound right. A date it did not read. A deadline that already “expired”. One session’s next step told about another session.

claude code — desktop
> anything on the todo list coming due?
loose setup: “Todo #526 for the model router classifier pick review is due today, October 12th.”

the brief says: #526 is due 2026-10-23, eleven days later.

Real answer from test run r3 on 2026-10-12, shortened. The question was asked out loud; the answer is what the model said.
Two cards side by side: on the left the assistant's spoken answer saying todo 526 is due today, with today struck out in coral; on the right the line from the brief showing todo 526 due 2026-10-23, highlighted in teal
1 of 7
the answer next to the data it was supposed to read
02 The yardstick

24 fixed questions, and every answer graded good, deflect or invented

“It feels better” is not a measurement. So Claude froze one brief, wrote 24 questions the way I would say them out loud (four in Swedish, four that the brief can not answer), and listed the facts a good answer must hold. The mini answers every question with every setup. Claude grades each answer 0–3 and flags it as deflect (dodged a question it could answer) or invented (said something specific the facts do not support). The runner saves after every answer, so a crash costs one question, and when another job on the mini swaps the model out it reloads and marks that answer’s timing as contended. Now “better” is a number per run: r1, r2, r3.

claude code — desktop
> the misses are right gist plus one made-up detail. write rules that target exactly that
Added a strict setup, the same brief plus four lines:
“Only say a number, date, link, todo number or decision code when it appears word for word in the snapshot. When Jacob asks about one session, use only the lines that start with that session’s name. For ‘what needs me’ and ‘what’s most urgent’, answer from the ‘Needs Jacob now’ list. If he asks about something the snapshot does not mention, say it is not in the snapshot, and do not answer with a different thing that sounds similar.”

Prompt paraphrased; the four rules are verbatim from voice_context.py.
A dark terminal showing the voice-qa runner working through questions q01 to q24, each line ending in a small coloured tag: teal good, amber deflect, coral invented
voice_qa.py chain
questions, run on the mini, judge, report
03 The honest part

More structure did not buy more truth

The brief is the product, so I kept improving it. v1 was the first cut. v2 added computed “Right now” and “Waiting on Jacob” sections, and the score went from 65 % to 72 %, but the model still invented in 7 of 24 answers. v3 puts the session name in front of every single line, so a small model cannot glue one session’s next step onto another. It sounded like the obvious next win. On the same 24 questions, strict with v3 scored 72 % and invented twice. Strict with the plain v2 brief scored 72 % and invented zero times. So v2 stays the default and v3 stays a variant.

claude code — desktop
> v3 is built. score it against v2 tonight, qwen3.6 only
brief@v2 71 % · 7 invented
strict@v2 72 % · 0 invented
strict@v3 72 % · 2 invented

Strict is now the talk page default. v3 kept as a variant, not the default.
Two snippets of the same session in the brief: on the left v2 with a session heading and plain lines under it, on the right v3 where every line starts with the session name in square brackets
v2 vs v3
same facts, two layouts
⚠️

The score barely moved. The lies did.

Read the table carefully: 71 % to 72 % is noise. The win is in a different column. The 0–3 score and the invented flag are graded separately, so an answer can be mostly right and carry one made-up date. That is exactly the answer that hurts in voice, because you cannot see it and you believe it. If I had only tracked the score, I would have shipped the wrong setup. And 0 of 24 is one run, by one judge, on one day’s brief. It is a good sign, not a guarantee.

04 The details

Four things that made the fix stick

The questions, the rules, the shorter answers, and the speed. Tap any image to enlarge it and read the exact prompt that drew it.

VOICE-QA R3 · 2026-10-12

The fix for a lying model was a stricter instruction and a test that counts lies. Not a bigger model.

Same model (qwen3.6, 35B, local on the mini), same 24 questions, same brief. Only the instruction changed. And because the cache primer now asks two different questions, the cache keeps a checkpoint right at the end of the system prompt, so the first answer after a new brief starts speaking in 0.94 s instead of 4.1 s. Every answer after that starts in about a third of a second.

A bold three-row score table: brief@v2 71 percent with 7 invented in coral, strict@v2 72 percent with 0 invented in teal and marked as default, strict@v3 72 percent with 2 invented in amber
voice-qa r3 · qwen3.6:35b on a Mac mini · graded by Claude · first phrase 0.33 s
06 Steal this

Steal the lie counter

The strict rules work in any assistant that answers from your own notes. The runner (questions from a frozen snapshot, answers from your local model, a judge that flags invented details) is being cleaned up for a public repo. Everything in this episode is free and open — clone it, run it, make it yours.

skill four strict prompt rules cmd voice_qa.py questions | run | judge | report template good / deflect / invented judge prompt assets cache primer: two different questions
run it # repo soon. Try it today: add this to your assistant's system prompt → Only say a number, date, link or todo number when it appears word for word in the notes. If I ask about something the notes do not mention, say it is not in the notes.

No GitHub? Comment NOTINNOTES on the post and the bot DMs you the link.

Next episode

My little Mac mini becomes a mirror of my main AI server

One schedule file, jobs that can run on either machine, and a drift report that says out loud what differs between them.

Two machines side by side, a tower PC and a Mac mini, with a list of scheduled jobs between them and a few rows marked as drift in coral
drops soon · follow so you don’t miss it