When a fine-tuned 4B model beats a general-purpose LLM
Background
At Alhena we ran a multi-agent, planner–executor system: a central agent broke each request into subtasks and handed them to specialized agents that plan, execute, and verify. The architecture was a big win for context management, each agent only had to hold the slice of context it needed. But the extra planning and verification calls added up, and our LLM spend rose by roughly 50%.
That sent me looking for a cheaper way. Before this, my working assumption had always been that bigger is better: more weights, more capability. But the most common decisions in our system, planner choices, context checks, and human-handoff calls, are narrow and repetitive. The question became concrete: for a specific, well-defined task, do we actually need a large general-purpose model, or can a small model trained on the right data do better?
Method
I fine-tuned an open 4-billion-parameter model, Qwen3-4B, on high-quality, domain-specific data from customers who had agreed to let us use it. The fine-tuned model was trained to reproduce three production decision types:
- Planner decisions, how to decompose a request and route it to agents
- Context checks, verifying that an action keeps the relevant context in view
- Human-handoff decisions, whether and when to escalate to a person
We evaluated the model two ways: on a labeled test dataset we held out from fine-tuning, and with production experiments against our existing baseline, a much larger general-purpose model. The headline metric was F1, since handoff and routing are imbalanced classification problems where accuracy alone is misleading:
A high F1 means the fine-tuned model both finds the right decision (recall) and rarely commits to the wrong one (precision), which is exactly what a production router needs.
Results
| Metric | Large general-purpose model | Fine-tuned Qwen3-4B |
|---|---|---|
| F1 (evaluation set) | 0.47 | 0.83 |
| Production latency | baseline | ≈30% lower |
| Inference cost | baseline | ≈40% lower |
The fine-tuned 4B model nearly doubled F1 against a model many times its size, while being faster and cheaper to run. That flipped my assumption on its head, and raised more questions than it answered: what changed in the weights, and why did a fraction of the parameters do so much better on this task?
Failure cases
This isn’t a universal win for small models. Places where the fine-tuned model struggled, or where I’d be careful before relying on it:
- Out-of-distribution requests. On queries that looked nothing like the fine-tuning distribution, confidence dropped and the model guessed. We gate low-confidence decisions to a human or a larger model rather than trust the small model blindly.
- Ambiguous and multi-intent messages. Requests that legitimately straddle two intents were hard for a narrow classifier, occasionally routing to the wrong agent.
- Label noise and leakage. Customer data is messy; early scores improved measurably after cleaning mislabeled examples and removing duplicates between train and eval.
- Domain overfitting. Performance was strongest on domains well represented in the training data. New verticals or customers can regress until the model sees more of their data.
What I’d test next
- Sweep model scale. Where is the crossover? Compare 1B, 4B, 7B, and 14B fine-tunes on the same data to find where the added capacity stops paying for itself.
- Isolate what matters. Ablate data quality, quantity, and training recipe (LoRA vs. full fine-tune, epochs, learning rate) to separate the effect of data from the effect of architecture and procedure.
- Measure per-class, not just aggregate. Move from a single F1 to per-intent precision/recall, plus human evaluation on handoff decisions where errors are expensive.
- Fine-tune vs. distill. Compare direct fine-tuning against distilling the large model’s behavior on the same tasks, does the small model inherit the gap, or close it?
- Confidence-aware cascades. Route confidently, fall back on uncertainty, small model first, large model on demand, human as the final gate, and measure end-to-end accuracy and cost.
- Generalization. Test held-out customer domains to understand when the small model’s advantage disappears.
Takeaway
The result isn’t that large models are obsolete. It’s that model selection should be driven by the task, not by a reflex toward scale. On a narrow, well-specified problem with good data, a small specialized model can be more accurate, faster, and far cheaper. The most interesting open question is why, and under what conditions that advantage holds. That’s what I want to keep studying.