LLMs have demonstrated remarkable performance with tremendous potential across a wide range of tasks. However, deploying these models has been challenging due to the astronomical number of model parameters, which necessitates large memory capacity and high memory bandwidth. In this chapter, we propose an effective approach to make the deployment of LLMs more efficient. We support automatic INT4 weight-only quantization and design a specialized LLM Runtime with highly optimized kernels to accelerate LLM inference on CPUs. We demonstrate the general applicability of our approach on popular LLMs, including Llama (Touvron et al. 2023a), Llama2 (Touvron et al. 2023b), and Mistral (Jiang et al. 2023), showcasing extreme inference efficiency on CPUs. The code is publicly available at: https://github.com/intel/intel-extension-for-transformers .