Although fire detection systems have advanced significantly, most existing approaches rely on large labeled datasets and extensive task-specific tuning, which limit flexibility and scalability. Effective fire detection requires the ability to identify fire in previously unseen scenarios, under diverse environmental conditions. In this paper, we present SpectroFire, a lightweight zero-shot fire detection system optimized for UAV-based real-time monitoring. SpectroFire employs a custom CNN encoder to transform RGB and IR inputs into compact embeddings, aligned with semantic prompts through a CLIP-style dual encoder trained with contrastive learning. Experiments on the FLAME 2 benchmark demonstrate that SpectroFire achieves both high accuracy and low-latency inference, validating its feasibility for scalable wildfire monitoring on edge devices. Furthermore, we integrate SpectroFire into a live UAV dashboard, confirming its practicality for real-world deployment in safety-critical environments.