Message classification before LLMs
Background
A decade before language models made auto-replies feel easy, businesses on Messenger faced the same problem in a cruder form: thousands of incoming messages and not enough people to answer them all. At Facebook, I worked with the AI research team to build auto-response generators for Pages, systems that read an incoming message, decide what the customer wants, and reply with the right next step.
There was no general-purpose model to lean on. The whole system had to be assembled from smaller, trainable pieces: a classifier that recognized intent, and a library of conversational workflows that turned a recognized intent into a useful reply. This was where I first learned how hard it is to train and tune a model well.
The system
The auto-response stack had three parts.
- Intent classification. A supervised model mapped each incoming message to one of a set of intents: order status, shipping, hours, refunds, and so on. The model was trained on labeled examples and tuned until it was confident enough to act.
- Conversational workflows. Each intent pointed into a scripted flow, a sequence of turns that gathered the missing details (order number, size, address) and emitted a response. The workflow was where the business logic lived.
- Rules for the long tail. The classifier handled the common intents; hand-built rules covered edge cases and forced a handoff when confidence was low.
The classifier and the workflows were tightly coupled. A correct intent with a clumsy workflow still produced a bad answer, and a great workflow never fired if the intent was missed.
What was hard
The hard part was not the happy path. Messages are short, misspelled, and full of context the words alone do not carry. “Where is it?” can mean order tracking, a store location, or a missing account feature. Getting a model confident enough to act, and humble enough to hand off, was a tuning problem more than a coding problem.
Every improvement seemed to trade one failure for another: raising the confidence threshold reduced wrong replies but routed more messages to humans; adding intents made the model more capable but also more likely to confuse neighboring intents. That tension, between capability and reliability, is the same one I still think about today.
Failure cases
- Ambiguous and multi-intent messages. “Can I return this and reorder?” spans two intents; a single-label classifier had to pick one and got it wrong often enough to hurt.
- Low-confidence routing. On unfamiliar phrasing the model guessed, and once a message was routed to the wrong workflow it was hard to recover mid-conversation.
- Cold-start intents. New kinds of requests had little training data, so early models performed worst exactly where businesses were adding new services.
- Cross-lingual and informal text. Abbreviations, typos, and mixed languages degraded classification in ways the training set did not anticipate.
- Brittle flows. Scripted workflows did not generalize across brands; each new business needed tuning to sound and behave like itself.
What it taught me
This work is the reason a much later experiment stopped me in my tracks. Years afterward, at Alhena, I fine-tuned a small model and watched it beat a much larger general-purpose model on our domain. The result did not feel like magic; it felt like the same lesson from Messenger, finally with better tools. That follow-up experiment is written up here ↗.
The deeper lesson is stable across both eras: the model is the easy part, and the data, the evaluation, and the knowledge of when to trust the model are the hard part.