Large language models can generate harmful and offensive content, particularly when responding to toxic prompts. Existing approaches to mitigate such content face significant challenges: fine-tuning requires extensive curated datasets, retrieval-based methods need comprehensive non-toxic references, and prompting techniques demand multiple inference rounds that increase computational overhead. In this work, we propose the expert-guided toxicity filtration for debiased generation (exposed) to eliminate the undesired harmful outputs for LLMs without the aforementioned requirements. exposed constructs a debiasing expert based on the abundant toxic corpus to identify and expose the potentially dangerous tokens. It then processes the output to the LLMs and constructs a fair distribution by suppressing and attenuating the toxic tokens. exposed is evaluated on fairness benchmarks over three LLM families. Extensive experiments demonstrate that compared with other baselines, the proposed exposed significantly reduces the potential social bias while balancing fairness and generation performance.