<p>This paper proposes De-Codec, a codec method that decouples speech and background-sound (BGS) tokens during the tokenization of input audio, enabling downstream audio tasks to explicitly select efficient input tokens. To achieve complete decoupling, we design a dual-path quantization block inserted into an Encoder-Decoder network for the independent vector quantization of speech and background sound. Moreover, we propose a token swapping training method, which replaces the BGS tokens of the input audio and supervises De-Codec to reconstruct the input speech with the replaced BGS. The experimental results demonstrate that the proposed De-Codec can achieve complete decoupling of speech and BGS tokens while maintaining audio reconstruction performance. Moreover, we implement the De-Codec on speech denoising and BGS replacement tasks, verifying the efficiency of the decoupled tokens for downstream audio tasks.</p>