Large language models (LLMs) have demonstrated remarkable performance across a diverse range of tasks, including natural language understanding, text generation, and more. However, their vast number of parameters and high computational demands impede their widespread deployment. Quantization technology stands as one of the most effective approaches to reducing the costs associated with the inference and deployment of LLMs. The existing W8A8 quantization method suffers from a significant performance degradation after model quantization due to the emergence of outliers in activations. In this paper, we put forward an Automatic Layer-by-layer Mixed-precision Quantization (ALMP) algorithm. We carefully construct a dataset based on the activation distribution characteristics and then train a high precision prediction model. Based on this model, ALMP determines whether a specific layer needs to be quantized, thereby achieving mixed-precision quantization of INT8 and FP16. Comprehensive experimental results demonstrate that the ALMP yields speedups of 1.4× and 1.2× in the pre-filling and decoding phases, respectively. Notably, for models with over 70B parameters, the quantization-induced accuracy degradation is less than 1%.