Small models, narrow jobs

The default is correct, until it isn’t

Reaching for the largest available model is the right decision while you are still deciding what the job is. It absorbs vague prompts, tolerates bad examples, and lets you find out whether the idea works at all without also running an optimisation project.

It is rarely the right decision afterwards. Once the job is fixed, narrow, and running ten thousand times a day — classify this, extract these six fields, rewrite this into house style — you are paying frontier prices for a task that has stopped being open-ended.

Where small models hold

In our testing so far the pattern is unsurprising but worth stating: the more the task looks like a function with a stable signature, the better a small or task-tuned model does. Fixed input shape, fixed output shape, a bounded vocabulary of decisions, and enough real examples to tune on.

• Classification into a closed set, especially with good label definitions.
• Field extraction from documents whose structure varies but whose fields do not.
• Reformatting and normalisation — the work that used to be a regular expression and a lookup table.
• First-pass triage where the expensive model is still available behind it.

Where they fail quietly

This is the part that matters, and it is why “swap in a smaller model and watch the metrics” is not a sufficient method.

Large models fail loudly. They hedge, they refuse, they tell you the input was ambiguous. Small tuned models tend to fail confidently and in-distribution: they return a plausible value in the right format that happens to be wrong. Nothing in the response indicates that anything went wrong, so nothing downstream catches it, and the error rate you measure is the error rate you can see.

A large model tells you it is unsure. A small one hands you a well-formatted mistake.

So the interesting question is not accuracy on the happy path. It is whether the small model can be made to express uncertainty at all, and whether that signal is good enough to route the hard cases upward.

An honest comparison

The failure mode of this kind of evaluation is that it is written by the person who wants the small model to win, usually because they proposed it. We are trying to design the comparison the other way round: the evaluation set is built by someone briefed to find the cases where the small model embarrasses itself, and cost is reported as total cost including the escalations, not as a price-per-token ratio.