For about six weeks a frontier AI model read a fixed script aloud to my journal, four times a day. What that taught me about where the model belongs, and why I did not train my own.
My day starts with a journal page: the date, the open tasks, the calendar, a summary of what arrived overnight. In March a scheduled Claude task built it four times a day, at 05:00, 09:00, 13:00 and 17:00, working through some seventeen numbered steps in a long instruction file: create the page from a template, tick off finished tasks, copy the calendar, sort the mail. It worked well, on the most capable model I could rent.
What the model was doing
On 21 March, the day of post 2’s restart note, I wrote a concept note marking which of those steps needed a language model at all. Most did not: filling a template, sorting a table and matching dates are plain programming. The assistant ran for five more weeks anyway. It worked, and replacing something that works is easy to put off.
In May Claude went through the whole record. It sorted 885 sessions from February to May into three tiers: plain Python would do (198), a small local model would do (401), or the job needed a frontier model (286). The method was rough, based on signals such as the first message, output volume and tool mix; I put it at about 15 per cent either way. On the scheduled assistant, about 160 runs, it was clear: “pure orchestration, stepping through a fixed script”.
The assistant itself had been retired on 25 April, before the analysis; Python took over its routines.
Three tiers
The rule I now apply to every new job: Python first, a local model second, a frontier model last, each only where the simpler one cannot do it. On 31 August I counted 67 scheduled jobs across my machines; four called a cloud model, about 6 per cent. Three of the four jobs still listed for migration no longer existed in that form; the move had happened piecemeal, as a side effect of other work.
Should I train my own model?
In July came the opposite temptation: training a model on my own data, six months of chat history and a vault of notes. Claude advised against it. Training teaches a model behaviour, not facts, and my vault is mostly facts, which belong in a search index. First, build a scoreboard.
Before any evidence existed I wrote down a decision rule: if five retrieved examples in the prompt were good enough, there would be no training, and that would count as a success. The scoreboard, an evaluation harness, used 100 bookkeeping entries whose category a person had corrected, and scored whether a model got area and category exactly right. qwen3, with eight billion parameters, got 22 per cent with no examples and 70 per cent with five similar entries from my own history. Eight examples gave the same 70. The larger gemma3, with twelve billion, scored worse and ran about nine times slower, because it did not fit the 8 GB graphics card.
Two things the scoreboard caught
Building it exposed two mistakes. Claude’s analysis had said that a particular database field marked human-confirmed answers. It marked none; the real signal was in the audit log, where a named person had corrected a category. The second was mine: many entries carried comments narrating the correction, so a model could have scored by reading the answer out of its input. Claude and I scrubbed them. Any system that logs its own edits can leak its answers this way.
The pattern
Python where predictability matters, a local model for routine judgement, a frontier model where judgement is the product — and measure before you train. One question settles both stories: does the job need a fixed rule, a routine classification, or something where quality shows? I had not asked it of the journal assistant; the scoreboard asked it of the training idea before I spent anything.
It is not fully settled. The 70 per cent leaves a tail, and some misses may be a ceiling of the input: an insurance debit could belong to cars or to travel. Interactive work (the morning and quarterly reviews, writing, design) stays on the frontier model on purpose.
Part 3 of twelve in the series on how I built my AI OS. Start with the preface.
Next: three machines and a laptop that sleeps — most of what went wrong in my first six months had one cause.
* Host names are changed (here: Yavin, my home server). The real ones come from a different saga.
Appendix — for builders
- List the steps of a job before choosing a model. Mark each: fixed rule, routine judgement, or judgement where quality shows. Often a regex, a small local model or a person reading the result is enough; keep the frontier model for design, debugging and writing.
- Try retrieved examples in the prompt before fine-tuning. Five gave me most of the gain; eight gave no more.
- Build the golden set from human corrections, not from model output; scrub anything that gives the answer away; keep the example pool apart from the entries you score.
- Write the decision rule before the result. It is hard to set once the effort is spent.
- Keep usage records. They let me classify 885 sessions after the fact.
- What I considered instead: everything on the frontier model (simplest and best quality, but you pay for plumbing and depend on the connection), a fine-tuned personal model (wrong for facts, worse for privacy), and everything local (cheap, but weaker where quality shows). I chose three tiers because each job then gets the cheapest thing that does it reliably.