Research note · Facebook (now Meta)

Message classification before LLMs

Auto-response generators for Pages with supervised intent matching

Background

A decade before language models made auto-replies feel easy, businesses on Messenger faced the same problem in a cruder form: thousands of incoming messages and not enough people to answer them all. At Facebook, I worked with the AI research team to build auto-response generators for Pages, systems that read an incoming message, decide what the customer wants, and reply with the right next step.

There was no general-purpose model to lean on. The whole system had to be assembled from smaller, trainable pieces: a classifier that recognized intent, and a library of conversational workflows that turned a recognized intent into a useful reply. This was where I first learned how hard it is to train and tune a model well.

The system

The auto-response stack had three parts.

The classifier and the workflows were tightly coupled. A correct intent with a clumsy workflow still produced a bad answer, and a great workflow never fired if the intent was missed.

What was hard

The hard part was not the happy path. Messages are short, misspelled, and full of context the words alone do not carry. “Where is it?” can mean order tracking, a store location, or a missing account feature. Getting a model confident enough to act, and humble enough to hand off, was a tuning problem more than a coding problem.

Every improvement seemed to trade one failure for another: raising the confidence threshold reduced wrong replies but routed more messages to humans; adding intents made the model more capable but also more likely to confuse neighboring intents. That tension, between capability and reliability, is the same one I still think about today.

Failure cases

What it taught me

This work is the reason a much later experiment stopped me in my tracks. Years afterward, at Alhena, I fine-tuned a small model and watched it beat a much larger general-purpose model on our domain. The result did not feel like magic; it felt like the same lesson from Messenger, finally with better tools. That follow-up experiment is written up here ↗.

The deeper lesson is stable across both eras: the model is the easy part, and the data, the evaluation, and the knowledge of when to trust the model are the hard part.