Trojan backdoors can be injected into large language models at various stages, including pretraining, fine-tuning, and in-context learning, posing a significant threat to the model’s alignment. Due to the nature of causal language modeling, the vast search space makes detecting these triggers challenging. This study proposes a multistage framework for detecting Trojan triggers in large language models consisting of token filtration, trigger identification, and trigger verification. Existing trigger identification methods are discussed, and two new variants are introduced. These novel trigger inversion methods rely on black-box assumptions, only requiring output logits, and utilize beam search and greedy decoding respectively. For verification, semantic-preserving prompts and character-level perturbations are used to differentiate between actual Trojan triggers and other adversarial strings that display similar characteristics. Finally, a poison detection score for the large language model under investigation is provided. The evaluation of the approach on the TrojAI and RLHF poisoned model datasets demonstrates promising results.