Research note · Alhena AI

When a fine-tuned 4B model beats a general-purpose LLM

November 2026 · San Francisco Bay Area

Background

At Alhena we ran a multi-agent, planner–executor system: a central agent broke each request into subtasks and handed them to specialized agents that plan, execute, and verify. The architecture was a big win for context management, each agent only had to hold the slice of context it needed. But the extra planning and verification calls added up, and our LLM spend rose by roughly 50%.

That sent me looking for a cheaper way. Before this, my working assumption had always been that bigger is better: more weights, more capability. But the most common decisions in our system, planner choices, context checks, and human-handoff calls, are narrow and repetitive. The question became concrete: for a specific, well-defined task, do we actually need a large general-purpose model, or can a small model trained on the right data do better?

Method

I fine-tuned an open 4-billion-parameter model, Qwen3-4B, on high-quality, domain-specific data from customers who had agreed to let us use it. The fine-tuned model was trained to reproduce three production decision types:

We evaluated the model two ways: on a labeled test dataset we held out from fine-tuning, and with production experiments against our existing baseline, a much larger general-purpose model. The headline metric was F1, since handoff and routing are imbalanced classification problems where accuracy alone is misleading:

\[ \mathrm{F1} = 2 \cdot \frac{\mathrm{precision} \cdot \mathrm{recall}} {\mathrm{precision} + \mathrm{recall}} \]

A high F1 means the fine-tuned model both finds the right decision (recall) and rarely commits to the wrong one (precision), which is exactly what a production router needs.

Results

Metric Large general-purpose model Fine-tuned Qwen3-4B
F1 (evaluation set) 0.47 0.83
Production latency baseline ≈30% lower
Inference cost baseline ≈40% lower

The fine-tuned 4B model nearly doubled F1 against a model many times its size, while being faster and cheaper to run. That flipped my assumption on its head, and raised more questions than it answered: what changed in the weights, and why did a fraction of the parameters do so much better on this task?

Failure cases

This isn’t a universal win for small models. Places where the fine-tuned model struggled, or where I’d be careful before relying on it:

What I’d test next

Takeaway

The result isn’t that large models are obsolete. It’s that model selection should be driven by the task, not by a reflex toward scale. On a narrow, well-specified problem with good data, a small specialized model can be more accurate, faster, and far cheaper. The most interesting open question is why, and under what conditions that advantage holds. That’s what I want to keep studying.