Model distillation: what it is, and why smaller AI models keep getting better

The problem smaller models are solving

Building a very large AI model is expensive. Training it takes enormous computing power, and running it afterwards, known as inference, costs money and electricity every single time someone uses it. A model with hundreds of billions of parameters might give brilliant answers, but it is slow, costly to host, and impractical to run on a laptop or a phone.

This is the gap that model distillation is designed to close. Rather than trying to build a smaller model from scratch and hoping it performs well, distillation takes a large, capable model and uses it to train a smaller one to imitate its behaviour. The result is often a compact model that captures a surprising amount of the bigger model’s ability, at a fraction of the size and running cost.

The teacher and the student

The standard way to explain distillation is through a teacher and student analogy, and it holds up well because it is essentially accurate.

The large, already-trained model is the teacher. Instead of only telling the smaller student model whether an answer is right or wrong, the teacher shares its full range of predictions, including how confident it was in the answers it rejected. This richer signal, sometimes called the teacher’s probability distribution over possible answers, gives the student far more to learn from than a simple pass or fail mark would.

The student model, which has far fewer parameters and is cheaper to train and run, is then trained to reproduce those patterns. It never sees the raw scale of the teacher’s internal workings, only its outputs across many examples, but that turns out to be enough to transfer a large share of the teacher’s competence.

This idea is not new to generative AI chatbots. It was described in machine learning research over a decade ago, originally applied to image recognition systems, long before large language models existed. What has changed is the scale at which it is now used, with distillation applied to some of the most capable language and image models in the world to produce smaller versions that businesses and developers can actually afford to deploy.

Why this is not the only trick in the box

Distillation usually sits alongside other compression techniques, and it is worth knowing the difference so the claims made about any given model make sense.

Quantisation reduces the precision of the numbers a model uses internally, similar to saving a photo at lower resolution. Pruning removes parts of a model’s internal connections that contribute little to its output. Distillation is different because it creates an entirely new, smaller model trained to mimic the larger one, rather than simply trimming or compressing the original. In practice, developers often combine all three, distilling a model down and then quantising it further for use on a phone or in a browser.

Why smaller models keep improving over time

It might seem like there should be a hard limit to how good a small model can get, since it has far less capacity to store information than a giant one. In practice, three things keep pushing that ceiling upward.

First, teachers keep improving. Every time a new frontier model is released, it becomes a better source of training signal for the next generation of distilled, smaller models. A student trained by a stronger teacher generally ends up stronger itself.

Second, training methods keep improving. Researchers have got better at deciding which examples matter most for distillation, how to structure the training process, and how to combine distillation with techniques like fine-tuning on specialised data. A smaller model trained well on the right data can outperform an older, larger model trained less carefully.

Third, architecture design keeps improving. The internal structure of models, not just their size, affects how efficiently they use their parameters. Innovations that let a model do more with fewer parameters, independent of distillation, mean that today’s small models are built on better foundations than yesterday’s.

Together, this means the size of a model is an increasingly unreliable guide to how capable it is. A well-distilled, well-designed small model can now match or beat a much larger, older one on many everyday tasks.

Why this matters beyond the tech industry

For ordinary users and UK businesses, this trend is what makes AI features usable on everyday devices, from smartphones to laptops, without needing a constant connection to a distant data centre. It also has an energy dimension: running a smaller model uses less electricity per query, which matters as data centre power demand becomes a bigger part of national energy planning and public debate.

There are trade-offs worth keeping in mind. A distilled model is fundamentally limited by what its teacher knew and how faithfully the training process transferred that knowledge, so it can inherit the teacher’s blind spots and biases as well as its strengths. Independent, up-to-date benchmarking is the only reliable way to judge how a specific small model performs on a given task, since marketing claims about efficiency and capability change frequently and should not be taken at face value.

For readers wanting to understand the policy and safety context around these models in the UK, the Department for Science, Innovation and Technology and the Alan Turing Institute both publish accessible material on how AI systems are evaluated and governed, and are a better source for current detail than any single manufacturer’s claims.