A Broader Study of Spectral Missing in Multi-spectral Vehicle Re-identification

Recent advances in vehicle re-identification based on complex illuminance rely on multi-spectral data: visible (VI) spectra, near-infrared (NIR), and thermal infrared (TIR) spectra. However, in various applications, the spectra are always partially missing in terms of the mismatch between the training data assumptions and the actual spectral types in the test data. Spectral missing becomes an open-world challenge for testing the robustness of models. However, only a few models have spectral missing accuracy tests. In this work, criteria for constant independent spectral missing, constant Siamese spectral missing, random individual spectral missing, random Siamese spectral missing, and random individual & Siamese spectral missing are established. Extensive experiments have been conducted on the proposed criteria to evaluate the accuracy difficulties of state-of-the-art one-stream learning and multi-stream learning in spectral missing. The result shows that the most advanced multi-stream learning performed better than the one-stream learning models. In some cases, the performance of multi-stream learning is even worse than that of one-stream learning methods in Siamese spectral missing. In the benchmark test, due to the complementarity of TIR and VI and the redundancy of NIR and VI, VI + TIR Siamese spectra (1.6%) have a more moderate loss of accuracy than VI + NIR Siamese spectra (2.8%). In the end, the accuracy of all models tends to favor the VI that is more relevant to open-world scenarios. This work validates the value of benchmarking to better represent the spectral missing diversity seen in open-world practice and to guide future research.