As language models continue to scale in parameter count and reasoning complexity, achieving efficient adaptation and interpretable inference has become a pressing challenge. Decomposition offers a versatile paradigm that addresses both concerns, enabling structural reduction during model training and logical segmentation during inference. This survey presents a comprehensive analysis of 29 decomposition methods categorized into algorithmic and contextual types. We examine how these methods restructure weight matrices, control update granularity, and decompose information content to support downstream tasks. Through extensive interpretations for each of these methods, we highlight the growing role of decomposition as both a mathematical and reasoning tool. Our findings offer a structured reference for advancing efficiency, modularity, and interpretability in large language models, with implications for research in model training, algorithm optimization, and knowledge-grounded inference. This work targets researchers and practitioners seeking scalable solutions across NLP domains where resource constraints and reasoning depth demand more than traditional modeling and inference pipelines.