Feature Selection and Year-Effect Dominance in FTIR Spectroscopy of Olive Oil: Implications for Cross-Year Predictive Stability
摘要
In multi-year agricultural monitoring, the reliability of FTIR spectroscopy is systematically undermined by the “year effect”: inter-annual environmental drift dominates spectral variance and can mask the biological treatment signal. We developed a Competitive Feature Selection (CFS) framework that uses dual competing Random Forest classifiers, one maximising treatment discriminability, one penalising year-discriminative wavenumbers, through a selectivity index computed within year-isolated nested tenfold cross-validation. CFS was applied to 324 olive-oil (Olea europaea L. cv. Picholine Marocaine) FTIR spectra collected across two consecutive harvest seasons under a factorial water–nutrient (NPK) design. Within a single season, treatment is highly classifiable (≈92% accuracy) and the two seasons are almost perfectly separable (year-classification ≈99%), confirming a dominant inter-annual drift. CFS yields a compact, directly interpretable 128-wavenumber subset, with stable selectivity rankings across folds (Spearman ρ̄ = 0.67), that recovers ~ 97% of full-spectrum in-distribution accuracy and is centred on the carbonyl (≈1745 cm⁻1) and aliphatic C–H (from 2800 to 3000 cm⁻1) regions. However, under leave-one-year-out validation, training on one season and testing on the unseen season, no method tested, including CFS, exceeded random chance for cross-year treatment prediction (≈12% for 9 classes; chance = 11.1%); manifold visualisation confirms that year structure persists in the CFS feature space. CFS is therefore best positioned as an interpretable feature-selection and year-effect-diagnosis tool rather than a solution to cross-year prediction: on two seasons, feature selection alone does not deliver year-invariant prediction, and genuinely year-robust FTIR monitoring will require additional source seasons, paired calibration-transfer samples, or out-of-distribution-validated domain adaptation.
Graphical Abstract