Abstract

The last five years have seen explosive progress in large language models (LLMs) — exemplified by systems such as ChatGPT and GPT-4 — which deliver broad capabilities but at heavy computational, latency, privacy, and cost budgets. In parallel, a renewed research and engineering focus on Small Language Models (SLMs) — compact, task-optimized models that run on-device or on constrained servers — has produced techniques and models that close much of the gap while enabling new applications (on-device inference, embedded robotics, low-cost production). This article/review compares SLMs and LLMs across design, training, deployment, and application dimensions; surveys core compression methods (distillation, quantization, parameter-efficient tuning); examines benchmarks and representative SLMs (e.g., TinyLlama); and proposes evaluation criteria and recommended research directions for widely deployable language intelligence. Key claims are supported by recent surveys, empirical papers, and benchmark studies.

1. Introduction & Motivation

Large models (billions to hundreds of billions of parameters) have pushed capabilities for zero-shot reasoning, instruction following, and multi-turn dialogue. However, their deployment often requires large GPUs/TPUs, reliable cloud connectivity, and high inference cost — constraints that hinder low-latency, private, and offline applications (mobile apps, robots, IoT). Small Language Models (SLMs) are intentionally compact architectures (ranging from ~100M to a few billion parameters) or compressed variants of LLMs designed for on-device or constrained-server inference. SLMs are not merely “smaller copies” of LLMs: the field now includes architecture choices, fine-tuning regimes, and tooling (quantization, distillation, pruning) that produce models tailored for specific constraints and use-cases. Recent comprehensive surveys document this growing ecosystem and its practical impact.

2. Definitions & Taxonomy

LLM (Large Language Model): Very large transformer-based models (≥10B params typical) trained on massive corpora. Strengths: generality, emergent capabilities. Weaknesses: cost, latency, privacy exposure.

SLM (Small Language Model): Compact models (≈10⁷–10⁹+ params) or aggressively compressed LLM variants that aim for high compute/latency efficiency while retaining acceptable task performance. SLMs include purpose-built small architectures (TinyLlama), distilled students (DistilBERT style), and heavily quantized LLMs.

Compression & Efficiency Methods: Knowledge distillation, post-training quantization (GPTQ/AWQ/GGUF workflows), pruning, low-rank/adapters (LoRA), and mixed-precision training.