A team spent $500 and three and a half days training a 9B open-weight model to read product listings. It beat five frontier models. Fermisense reports that the best frontier setup reached 76.9% of the achievable score, and its trained 9B reached 87.3%. At $0.50 per 1,000 listings, that's 40 times cheaper than the cheapest frontier setup and about 340 times cheaper than the priciest.
Fermisense sells this kind of training, graded its own work and tested one job. We'd treat it as a lead worth testing, and you should too. We think the shape holds. The pattern also breaks fast off the job: an 8B tool-calling model that nearly matches Claude Opus 4.5 on a fixed set of single-turn calls scores 6.5 to Opus's 84.5 on web search.
Our claim, after reading more than 100 papers, model cards and engineering posts in full, and scanning 254 posts from the last four months: a small open-weight LLM, taught by a big model or a good test on a single workflow, can beat frontier models at that workflow for a fraction of the cost and energy. Off the job, it loses almost everywhere, so the big model keeps its seat.
In most of what we read, the teacher is a big model that writes the training data, grades the attempts or sets the bar. In Fermisense's run it was a scoring rule. Their playbook still starts with the big model, because "every call generates the data (inputs, decisions, corrections) that a specialist model later trains on."
So we think useful AI looks like a team. Cheap specialists do the repetitive work, a router decides who gets each request, and a big model stays on call for the hard, new questions.
How does a big model teach a small one?
Distillation is how a small model, the student, learns from a big one, the teacher, a bit like you learned from a good senior engineer. What matters is what you let the student copy.
Picture a label on your photo that says "this is a BMW." The teacher's full list of odds says more: a BMW looks a bit like a garbage truck and far less like a carrot. Hinton and colleagues made that point in 2015. For text, the list is the teacher's odds for every possible next word. (The raw scores behind those odds are called logits.) So your student gets a hint at every word.
The big labs build their small models this way, and you can copy the method. In Google's own small test, Gemma 2's 2B student, taught by a 7B teacher, averaged 67.7 on three benchmarks. The same 2B trained from scratch scored 60.3.
You can teach in three ways. The third needs no big teacher.
Copy. The teacher writes answers and your student trains on them. This is plain supervised fine-tuning (SFT). It's cheap. But your student only sees roads the teacher walked, and that's a narrow map.
Grade. Your student writes its own answer and the teacher scores every word. That's on-policy distillation: "on-policy" means learning from your own attempts.
Test. A unit test or a judge scores the whole attempt, and that score is the reward. The loop is reinforcement learning (RL). No big teacher is needed, only a scorer.
Pick by what you have. With a teacher's answers, you copy. With an open-weight teacher you can run (grading needs its per-word odds on text you hand it, which frontier APIs don't give), grade. With a test or a judge, reward.
Grading is our favourite of the three. The student writes the answer, which is cheap. The teacher only has to read it once, in a single forward pass through the big network. Qwen reports Qwen3-8B hit 74.4 on a math test after 1,800 GPU hours of this, against 67.6 after 17,920 hours of plain RL. That's math and code only, and it leaves out what you'd spend building the teacher.
Tests work too, and you may already have them. Microsoft trained FrogNano, a 4B coding agent, with AI-written tests as the grader and no stronger model's answers. Its score on SWE-bench Verified (500 real GitHub issues to fix) went from 39.4% to 61.5% over five rounds of training.
We'd hold that number loosely. That set doubled as its validation set, the yardstick it was tuned against. On benchmarks it never trained against, it scored 37.6% on SWE-bench Pro (harder, longer coding tasks) and 31.1% on Terminal-Bench 2.0 (work in a command line). We'd quote those.
Can a small model beat a big one?
Yes, at one job, and we have receipts. Phi-4, a 14B model, beat the model that taught it. Microsoft trained it mostly on GPT-4o's synthetic data, and it scored 56.1 on a graduate-level science test against GPT-4o's 50.6. It also won on competition math, 80.4 to 74.6, though GPT-4o still led on MMLU (a broad multiple-choice knowledge test), 88.1 to 84.8, so it wasn't a sweep.
This is the table for a skeptic. Read the last column first, because that's where most of these claims shrink.
| Who, job | Small vs big | Catch |
|---|---|---|
| Fermisense Listings | 9B 87.3% vs 76.9% | Vendor, own test |
| OpenPipe | 14B beat o3, 64x cheaper | Synthetic data |
| Microsoft Science | Phi-4 56.1 vs 50.6 | GPT-4o won MMLU |
| Predibase 31 tasks | 10 points over GPT-4 | GPT-4 won 6 |
| Hsieh et al. Benchmark | 770M beat 540B | Few-shot baseline |
OpenPipe's 14B Qwen email specialist answered inbox searches 5 times faster and 64 times cheaper than o3, and training took under a day on one H100, about $80 for the final run (failed runs, data generation and engineering time not counted).
In a second OpenPipe test, a customer-support specialist scored 93% against o3's 50%. The untrained Qwen base scored 62%, so we read the gap as house rules o3 didn't know.
Not every result we found beats the frontier. In one paper we liked, a tuned 8B reached 87 to 98% of Claude Sonnet 4.5's quality on two fixed workflows, at about 300 to 460 times lower cost per conversation, and stayed below it overall. Setup cost $50 to $80, and it paid for itself within 500 conversations in every workflow they tried. That's a good deal on a job you run a million times.
Where do small models fall down?
Everywhere else. Start with the skeptics' best case, because we agree with half of it. A 2023 Berkeley paper found that imitation models are "adept at mimicking ChatGPT's style but not its factuality". They fooled human raters, then closed little to none of the gap on tasks outside the training data. Those were 2023 base models, and later work has improved. Still, test your student on cases outside its training mix before you trust it.
The same paper holds the hinge of our argument. Copying one narrow behavior worked far better than copying everything. That's why we say "one workflow."
Breadth is where small models collapse, and you can see exactly where. On the Berkeley function-calling leaderboard (BFCL, April 2026), xLAM-2-8b scores 84.58 on the fixed single-turn questions, close to Claude Opus 4.5's 88.58, and it even edges Opus on multi-turn. On web search it scores 6.5 to Opus's 84.5.
In Google's own table, the most candid we found, Gemma 4's E4B scores 42.2% on Tau2, a test of agents using tools in simulated support chats, where the 31B scores 76.9%. The small one still beats the older 27B Gemma 3's 16.2%, so a new small model can top an old big one, though on breadth we wouldn't bet on it against a new one.
The teaching can fail too. We'd watch for three things.
-
A better teacher can make a worse student. Apple's team saw this capacity gap: past a point, a stronger teacher hurts. Another paper found that models of 3B or smaller don't reliably gain from long reasoning chains, and a 2026 re-check says the gap doesn't always dominate. Test your own pair.
-
RL students find loopholes. OpenPipe's email agent learned to repeat its last tool call until it ran out of turns, because partial credit paid for turns.
-
New skills can erase old ones. Fine-tuning on new data made Qwen3-8B forget instruction following, even with LoRA (the cheap fine-tuning method in rule 4 below). Distilling from the earlier version nearly fixed it.
In what we read, copying rarely lifts a student past its teacher. Phi-4 did it, but Microsoft credits methods that go "beyond distillation". DeepSeek, which distilled R1 into smaller Qwen and Llama models, says going "beyond the boundaries of human intelligence" may still need stronger base models and larger-scale RL.
Distillation moves a skill into a smaller box. It rarely invents one.
Why do AI agents suit small models?
Because so many agent calls are boring, and boring is what a small model does well. Parse this. Pick a tool. Fill in the arguments. Open a few of your own traces and count them.
The authors of one position paper estimated, by examining how each agent calls its LLM, that about 60% of MetaGPT's calls could go to specialized small models. Open Operator came in at 40% and Cradle at 70%. They estimated it rather than measured it, and they'd keep a big model initially for architecture and debugging, which is where we keep ours.
Stanford's headline number has a trap in it. Across 1 million real-world chat and reasoning queries, local models answered 88.7% well enough, but only when the best local model for each query was picked with hindsight. Those were single-turn queries, with models of up to 20B active parameters (the part of a model that runs for each word).
The best single local model per year went from covering 23.2% of queries in 2023 to 71.3% in 2025. And it's uneven: the best single local model handled 88.9% of chat questions but only 64.9% of reasoning ones. So the router's real job is spotting the reasoning-heavy question before a small model fumbles it. Ours uses a 0.5 floor, set by hand and never tuned on labelled routes: when Jev is less sure than that, the main model gets the message.
What does a team of small experts look like?
It looks like a small conductor with big models on call. NVIDIA's ToolOrchestra trained an 8B conductor that scored 37.1% on the text-only part of Humanity's Last Exam (very hard expert questions) against GPT-5's 35.1%. The conductor calls GPT-5 as one of its tools. Hand the untrained 8B the same toolbox, GPT-5 included, and it scores 30.6, so training added the last 6.5 points. The authors' explanation: prompted models lean on themselves, or on the strongest tool "regardless of the cost".
Stanford's Minions pairs a local 8B with GPT-4o. It recovers 97.9% of GPT-4o's accuracy at 5.7 times lower cloud cost, on long-document tasks. Drop to a 3B and it still gets 93.4%. Drop to a 1B and it gets about half. Somewhere between 1B and 3B, the helper stops helping, and we'd test that floor before picking a size.
Another design escalates step by step, only on doubt, and its calls to the big model track how hard the task is. R2V needed the big model for 0.60% of decisions on HumanEval+ (small coding puzzles), where it scored 94.3%. On a terminal benchmark it needed it for about a third, and on a text game for 42%, on average.
On SWE-bench Pro, a tuned 4B helper that runs terminal commands saved its Claude Opus lead as many tokens (the word pieces models read and write) as an Opus helper did (12.7% against 12.8%) and solved about as many tasks (31.5% against 32.0%). The difference is whose tokens you pay for.
An easy request goes to the Extract specialist, as you'd hope. It's narrow, cheap and checkable.
An unsure router sends the request to the big model, so you pay for the frontier only where a small one would likely fail.
A failed check also sends it up. The tool call didn't pass your test, so the big model takes the step and the run goes on.
We'd keep one conductor in charge and never let the specialists talk to each other, because chatty agent teams fail a lot. Berkeley researchers found 41% to 86.7% failure rates across seven open multi-agent frameworks on 2024 and 2025 models, Anthropic says multi-agent systems use about 15 times the tokens of a chat, and Cognition argues that parallel agents make fragile systems when they don't share context.
How we route between models on Definable
We run a small version of this. On Definable, an agent can have a main model and an optional fast one. For each message, Jev, a cheap decision model, picks which of the two answers, and it must pick the fast one at 0.5 or more. If it's unsure, errors, runs slow or the fast model can't take the chat, the main model answers. When the router fails, you still get an answer. We built it to fail open.
Here's what it did on our development deployment (not production) from October 20 to 30: 570 routed messages across 10 workspaces. Jev sent 252 of them (44%) to the fast model and picked the main model for 195.
Another 105 went to the main model because Jev leaned fast but was less than 0.5 sure, and 18 because the chat was too long for the fast model's window. None fell back for an error or a timeout. Jev's median answer took 180 milliseconds and its slowest 405, well inside the one-second limit.
How much energy and money does a small model save?
Most of it, on paper. Three rows from the ML.ENERGY leaderboard, measured on NVIDIA B200 data-centre GPUs, show the spread. They differ in GPU count, precision and response length, and the board ranks energy alone, so don't read it as a winner list.
| Model | GPUs | Joules per response |
|---|---|---|
| GPT OSS 20B | 1 B200 | 61 |
| Qwen 3 8B | 1 B200 | 1,398 |
| DeepSeek R1 | 8 B200 | 26,820 |
Row two surprised us: the dense 8B (every parameter works on every word) uses 23 times more energy per answer than the 20B. Per token the gap is about 6 times: partly because gpt-oss-20b uses only about 3.6B parameters per token, partly precision (4-bit numbers against 16-bit ones) and how each was served. The other factor of about 4 is length, since the 8B's answers ran nearly 4 times longer.
So don't trust the size on the box.
Money tells the same story. Our own catalog lists Claude Opus 5.5 at $20 per million output tokens and gpt-oss-20b on OpenRouter at $0.09, about 222 times less. Those are list prices. What counts is cost per solved task, and if your small model needs retries, you give some back.
One test of 16 open models we read found a 26B open model about 15 times cheaper per passed task than GPT-5 on an owned Mac Studio. On rented cloud GPUs the gap fell to about 3 times. That was 30 synthetic tasks.
Does a local LLM save energy?
Mostly because it's small, not because it's local. The Stanford team behind the query numbers found a B200 data-centre GPU is 1.4 times better per watt than an M4 Max laptop chip on the same models. In their simulation, an 80%-accurate router cut energy 64.3% against sending every query to a 235B model in the cloud.
Now for what cuts against us. Reasoning models write 10 to 100 times more output tokens than chat models. And total use keeps climbing. The IEA counts 415 TWh for data centres in 2024 and projects about 945 TWh by 2030, and Jegham and colleagues warn that efficiency gains "do not inherently reduce overall environmental impact."
Small models make each answer cheaper. Cheaper isn't smaller. If cheaper answers bring ten times the traffic, a tenfold saving buys you nothing.
How do you train a small model for your workflow?
Six rules, in order.
-
Log the big model first. You can start this part today. The position paper recipe is to log every agent call that isn't a chat with a person, cluster them and tune one small model per cluster, with 10,000 to 100,000 examples as a rule of thumb. OpenAI offers the same idea as stored completions.
On Definable we already save every run as a trace, with each model call, each tool call and its cost, so the log exists before anyone asks for it. We don't train or host small models today.
-
Pick a step that's frequent and checkable. Our test is whether a script could score every outcome. Fermisense's checklist puts it as "every outcome can be checked by a rule, test, or rubric, with no person in the loop." Volume matters too, because that's where errors and costs compound.
-
Write the eval before training. We'd hold out cases the student never sees and score the big model on them first, so there's a bar to clear. Fermisense's trained 9B beat its own untrained base (64.2%) by 36%, a claim nobody can make without measuring the base first.
-
Teach in order. Copy first, with LoRA, which trains a small add-on instead of every weight. The original paper cut trainable parameters by 10,000 times on GPT-3, and Thinking Machines found it matches full fine-tuning at normal data sizes if a few details are right, though on code and math it learns less than full fine-tuning. Finish with grading or RL once the eval is trustworthy. RL also forgets less than SFT.
Then watch for loopholes. LLM judges have position, verbosity and self-enhancement biases a student can learn to exploit, and OpenPipe's agent already found the repeated-tool-call trick above.
-
Route small first, big on doubt. We'd send every request to the specialist first. Anything it's unsure about, or that fails a check, goes up to the big model.
-
Read the teacher's terms. OpenAI's terms say not to "use Output to develop models that compete with OpenAI", with exceptions in its services terms for classifiers kept in-house and for tuning OpenAI's own models. Anthropic's commercial terms bar building a competing product, "including to train competing AI models," without approval. Google's Gemini terms say not to "develop models that compete with the Services." The labs enforce it, too: in February 2026, Anthropic reported over 16 million exchanges with Claude run through about 24,000 fraudulent accounts.
"Compete" is the word we read twice. Open teachers are simpler. Qwen3-8B and gpt-oss-20b are Apache-2.0, and DeepSeek-R1 is MIT. Llama 3.1's license adds a naming rule, including "Llama" at the start of the name, plus a "Built with Llama" display rule. We aren't lawyers. Ask one.
For us it's a loop, not a project.
Log. The big model does the work for now. You keep every call: the input, the output and the cost.
Teach. Train the small one on those calls, then score it on held-out cases against the big model's score.
Route. You send requests to the specialist first. When it's unsure or a check fails, the big model answers.
Repeat. Your doubts are your next training data, so your specialist covers a bit more each round.
Which open models and tools should you start with?
Pick by the job and test two before committing. Every number below comes from the vendor's own model card, so rerun it on your data. Granite 4.2, gpt-oss-20b and Ministral 3 are Apache-2.0. Liquid's LFM3 uses its own "lfm1.0" license, so read it before you ship. If we had to start with one, we'd take Qwen3.5-9B: it's Apache-2.0, and it's the base behind Fermisense's 87.3%.
| Model | Size | Vendor's number |
|---|---|---|
| Qwen3.5-9B | 9B | 81.7 on a hard science test, over gpt-oss-120B's 80.1 (Qwen's table) |
| Gemma 4 E4B | 4.5B effective, 8B total | 42.2% on Tau2, 2.6 times Gemma 3 27B's 16.2% |
| LFM3-1.7B | 1.7B | 71.39 on function calling, over Qwen3.5-9B's 62.83 (Liquid's run) |
| Granite 4.2 8B | 8B | 52.39 on function calling (BFCL), about the same as the 3B |
| gpt-oss-20b | 21B, 3.6B active | Runs in 16GB of memory |
| Ministral 3 8B | 8B | Fits in 12GB of GPU memory at 8-bit precision |
For training, we'd begin with TRL: its DistillationTrainer runs the grading loop, matching a teacher's next-word odds on the student's own answers. We'd run the student locally with Ollama and serve many LoRA add-ons from one base with vLLM.
Our one strong opinion here: skip anything frozen. torchtune is no longer actively maintained, and the LoRAX and S-LoRA repos are archived. TRL's GOLD trainer, which learns from a teacher that splits text into different pieces (a different tokenizer), is still experimental, and TRL says its APIs "may change without notice." Unsloth says it trains 2x faster with 70% less GPU memory, a claim we haven't checked.
| Job | Tools |
|---|---|
| Train | TRL, Unsloth, OpenPipe ART |
| Serve LoRA add-ons | vLLM (per request) |
| Run locally | Ollama |
What we'd do on Monday
Pick one step in your agent that's frequent, checkable and boring. Log the big model doing it, so you have data. Write your eval, train the small one, and let the big one catch whatever it can't. You'll learn more from that than from any benchmark we quoted. On Definable, every run already keeps that log as a trace.
The big model isn't going away. It's just not your first call anymore.