Existing defenses against backdoor threats from third-party models primarily rely on trigger inversion-based defense. However, most methods impose trigger size constraints during trigger inversion, making it difficult to detect larger triggers. Additionally, these methods often use non-essential criteria to distinguish between natural and backdoor triggers and most of them use outlier detection methods that cannot detect many backdoors, leading to lower detection accuracy. This paper proposes a novel approach based on trigger inversion without trigger size constraints, using semantic consistency. We then train a detection model called InitDistillNet, which leverages the properties that backdoor triggers are ineffective in clean models and natural triggers are effective in similar models to accurately distinguish between backdoor triggers and natural triggers. The inverted triggers are verified for their effectiveness through InitDistillNet, and based on this verification, the backdoor triggers are detected and ultimately removed. Experimental results show that our method excels in scenarios with numerous backdoors, detecting larger triggers and achieving significantly higher detection accuracy than existing methods.