A Comprehensive Survey of Lightweight Large Language Models: Architectures, Techniques, and Challenges

The standard large language models (LLMs) mostly deliver strong performance across different NLP tasks, but tend to suffer from inference latency, memory footprint, and energy consumption, which may prevent deployment in resource-constrained and privacy-sensitive settings. The lightweight LLMs, on the other hand, can address these constraints by reducing the training time and storage cost while preserving the quality through some techniques like transfer learning (knowledge transfer from larger teacher models), model weights or architecture compression via quantization and pruning, parameter-efficient adaptation methods, inference-time acceleration methods, and hardware-aware deployment optimization. This survey made a systematic analysis of these techniques, clarifies the possible trade-offs, and summarizes the evaluation practice for lightweight LLMs by considering standard metrics (e.g. accuracy and generation quality), efficiency metrics (e.g. model size, latency, throughput, and memory), and responsible-AI criteria (such as robustness, bias, and safety). The research surveys and organizes recent lightweight LLMs and frameworks by comparing their architectures, training data, hardware requirements, and outcomes. Finally, the challenges, standardized reporting, multilingual coverage, safety under compression, and reproducibility are outlined with practical guidance for how to select the appropriate lightweight LLMs for real-world deployment.