Distillation
In plain terms
Training a small model to mimic a larger one.
Going deeper
Knowledge distillation transfers a large 'teacher' model's behavior into a smaller 'student' model, keeping much accuracy at lower cost. It compresses capability.
Under the hood
The student learns from the teacher's soft outputs; enables efficient edge and mobile deployment.
Examples
Model compression, edge deployment, faster inference, DistilBERT, efficiency.
Think of it like...
Like an apprentice learning to replicate a master's results far more cheaply.