What is Small language model?

How it works

An SLM uses the same transformer design as a large language model, just with fewer parameters. There is no official cut-off; "small" is relative and keeps moving as hardware improves. The term usually covers models that fit on consumer devices, especially after quantization.

Small models get good mainly through better data, not size. Microsoft's Phi-3 report, for example, attributes its small model's performance largely to heavily filtered web data and synthetic training data, and shows a 3.8-billion-parameter model running on a phone. Other common techniques are distillation (training the small model to imitate a larger one) and training on far more tokens than older scaling rules suggested for that size.

Why it matters

• Cost and speed: far cheaper per request and lower latency than frontier models.
• Privacy and offline use: they run on your own laptop, phone or server, so data stays local.
• Good enough for many jobs: classification, extraction, routing, summarising, autocomplete and simple tool calls.
• Agents: an NVIDIA research position paper argues that most repetitive sub-tasks in agent systems can be handled by small models, with large models reserved for the hard steps.

The limits are real: less world knowledge, weaker multi-step reasoning, shorter useful context and more hallucination on open-ended questions. Fine-tuning an SLM for one narrow task often closes much of the gap for that task.

Example

A company routes every incoming support email through a small open-weight model running on one GPU. It labels each email by topic and urgency and extracts the order number, which takes a fraction of a second per email. Only the complicated cases are passed to a larger, more expensive model to draft a reply.