<p>Spectral fluorescence lifetime imaging (S-FLIM) simultaneously deconvolves signal from multiple fluorophore species by leveraging both spectral and lifetime information. However, existing analyses still face multiple hurdles in decoding information collected from typical S-FLIM experiments. These include: using information from pre-calibrated spectra in environments that may differ from the cellular context in which S-FLIM experiments are performed; limitations arising from overlapping spectra; high photon budget requirements, typically about a hundred photons per pixel per species. Yet information on the spectra themselves are already encoded in the data and do not require pre-calibration. Moreover, efficient photon-by-photon analyses are possible reducing both the required photon budget and making it possible to use larger budgets in order to discriminate small differences in spectra to resolve spatially co-localized fluorophore species. To achieve this, we propose a framework, Bayes-S-FLIM, capable of simultaneously learning spectra and lifetimes photon-by-photon, using limited photon counts as low as 5000 photon to distinguish 3 species achieving high data efficiency. We demonstrate the proposed framework using synthetic and experimental data and show we can operate in limiting photon regimes and distinguish lifetimes with sub-nanosecond differences. Our synthetic data analysis suggests that we can deconvolve up to 9 species with heavily overlapped spectra.</p>