MSPD: Large Language Model Parallel Speculative Decoding Method Based on Multi-level Early Exiting

Speculative decoding (SD) is a method that utilizes an additional draft mode to predict multiple future tokens, which are subsequently verified by the target large language model (target LLM). Despite the significant acceleration achieved by SD, several challenges remain, including the training cost of the draft mode, drafting latency, and alignment and asynchrony issues with the target LLM. To address these challenges, we propose a novel SD model called MSPD (Multi-level Speculative Parallel Decoding). MSPD is composed of two main modules: first, the multi-level early-exit model, which treats the single-level early-exit model as a new target LLM and applies the early-exit model again for drafting. This approach addresses the training cost and drafting latency issues of the draft model and enhances its alignment with the target LLM by reusing the parameters of the target LLM. Second, the parallel inference model, namely Hypothesis Verification - Continuous Draft. This model enables parallel inference between the draft mode and the target LLM, avoiding the mutual waiting that causes synchronization bottlenecks and thus reducing the impact of asynchrony. Experimental results demonstrate that MSPD significantly accelerates the inference process while maintaining text generation consistency. Compared with the SOTA model, the number of parameters is reduced by 77.4%, and the inference speed on Spec-Bench is increased by 1.73 \(\times \) .