<p>Fire safety urgently requires better automatic early fire detection. While vision-based methods are promising, a clear benchmark for deep learning models tailored for this specific area has been lacking. This paper presents the first comprehensive vision-based benchmark of 33 deep learning models explicitly for automatic fire detection. The key novelty is the creation and utilization of a unique, real-world thermal infrared (IR) dataset derived from controlled room fire experiments by NRC Canada. This challenging dataset includes imagery of early-stage and fully developed fires, as well as variations from different test conditions. To assess broader applicability, model generalization was also evaluated using a general dataset (used in pre-training). By rigorously testing these models on both specialized and general datasets using multiple performance metrics (accuracy, speed, reliability, generalization, computational cost), this work establishes the first dedicated benchmark for deep learning in vision-based fire detection. This benchmark provides a novel and crucial resource for researchers to make informed decisions when selecting deep learning models for their specific fire detection applications, ultimately aiming to accelerate innovation and the development of more effective and reliable vision-based fire safety systems.</p>