Image memorability (IM) estimation typically relies on learning generic semantic features from large-scale datasets; however, memorability is intrinsically individual-dependent and personalized visual viewing behavior, but overlooking this variability can undermine model performance in downstream cognitive and human–machine interaction tasks, leading toward suboptimal performance. To address this, we propose PerMem, a unified framework that integrates knowledge distillation (KD) and imitation learning (IL) to jointly learn generic and personalized salient representations for memorability estimation by leveraging both image content and gaze-derived heatmaps. PerMem employs a teacher network built upon a pre-trained ResNet-50 backbone, followed by an encoder–decoder architecture coupled with spatial and channel attention mechanisms to generate generic saliency-aware memorability maps. A lightweight, attention-guided student encoder–decoder is then optimized through a composite imitation-guided distillation process, where knowledge is distilled from the teacher while simultaneously imitating user-specific gaze fixation heatmaps. Through this joint training process, the student network learns to produce personalized memorability estimates while achieving substantial reductions in computational complexity. We validate PerMem on two public IM estimation datasets (LaMem and SUN) and a in-house WoM dataset comprising of 45 participants (UMBC IRB #670) engaged in visual search and navigation tasks that reflect individualized visual attention patterns. PerMem outperforms nine state-of-the-art IM models by capturing coarse-to-fine saliency and adapting to individual attention, achieving <InlineEquation ID="IEq1"> <EquationSource Format="TEX">$\approx $</EquationSource> <EquationSource Format="MATHML"><math> <mo>≈</mo> </math></EquationSource> </InlineEquation>6% improvement in memorability prediction. Lastly, we evaluate PerMem on heterogeneous embedded edge devices, including Jetson Nano, Jetson Xavier NX, and Raspberry Pi, demonstrating efficient on-device inference with consistent reductions in memory usage (25.6–33.4%), power consumption (19.0–25.0%), and inference latency (34.9–39.8%) relative to the teacher model, highlighting its practical robustness for resource-constrained, human-centric applications.

Efficient personalized image memorability via gaze-guided semantic cloning distillation

期刊论文
出版模式：

开放获取
发表日期：

2026年7月7日

查看全文

Indrajeet Ghosh,
Mohammad Saeid Anwar,
Kasthuri Jayarajah,
Nirmalya Roy

摘要

Image memorability (IM) estimation typically relies on learning generic semantic features from large-scale datasets; however, memorability is intrinsically individual-dependent and personalized visual viewing behavior, but overlooking this variability can undermine model performance in downstream cognitive and human–machine interaction tasks, leading toward suboptimal performance. To address this, we propose PerMem, a unified framework that integrates knowledge distillation (KD) and imitation learning (IL) to jointly learn generic and personalized salient representations for memorability estimation by leveraging both image content and gaze-derived heatmaps. PerMem employs a teacher network built upon a pre-trained ResNet-50 backbone, followed by an encoder–decoder architecture coupled with spatial and channel attention mechanisms to generate generic saliency-aware memorability maps. A lightweight, attention-guided student encoder–decoder is then optimized through a composite imitation-guided distillation process, where knowledge is distilled from the teacher while simultaneously imitating user-specific gaze fixation heatmaps. Through this joint training process, the student network learns to produce personalized memorability estimates while achieving substantial reductions in computational complexity. We validate PerMem on two public IM estimation datasets (LaMem and SUN) and a in-house WoM dataset comprising of 45 participants (UMBC IRB #670) engaged in visual search and navigation tasks that reflect individualized visual attention patterns. PerMem outperforms nine state-of-the-art IM models by capturing coarse-to-fine saliency and adapting to individual attention, achieving $\approx $ $\approx$ 6% improvement in memorability prediction. Lastly, we evaluate PerMem on heterogeneous embedded edge devices, including Jetson Nano, Jetson Xavier NX, and Raspberry Pi, demonstrating efficient on-device inference with consistent reductions in memory usage (25.6–33.4%), power consumption (19.0–25.0%), and inference latency (34.9–39.8%) relative to the teacher model, highlighting its practical robustness for resource-constrained, human-centric applications.