<p>Nonlinear optical phenomena such as parametric amplification and frequency conversion are typically driven by external optical fields. Free electrons can also act as electromagnetic sources, offering unmatched spatial precision. Therefore, combining optical and electron-induced fields via the nonlinear response of material structures holds potential for revealing new physical phenomena and enabling disruptive applications. Here, we theoretically investigate wave mixing between external light and the evanescent fields of free electrons, giving rise to inelastic photon scattering mediated by the second-order nonlinear response of a specimen. Specifically, an incident photon may be blue- or red-shifted, while the passing electron correspondingly loses or gains energy. These processes are strongly enhanced when the frequency shift matches an optical resonance of the specimen. We present a general theoretical framework to quantify the photon conversion probability and demonstrate its application by revealing far-infrared vibrational fingerprints of retinal using only visible light. Beyond its fundamental interest, this phenomenon offers a practical approach for spatially mapping low-frequency excitations with nanometer resolution using visible photon energies and existing electron microscopes.</p>