Scholarlink

<p>Foundation models produce high-dimensional embedding representations whose geometric properties fundamentally determine downstream task effectiveness. Embedding anisotropy—the concentration of representations in a narrow cone—has been documented in natural language processing but not systematically studied in vision foundation models. We characterize anisotropy in embeddings from eight architecturally diverse models (ResNet50, DINOv2-S/L, BiomedCLIP, PLIP, SigLIP, UNI, and Virchow) across six MedMNIST benchmarks and two native-resolution clinical datasets (NCT-CRC-HE-100K and ISIC 2019; <i>N</i> &gt; 110,000 images total). All models exhibited substantial anisotropy: effective dimensionality ranged from 0.7 to 7.1% of the nominal embedding dimension. Post hoc whitening (PCA/ZCA) reduced classification AUC in 34 of 36 model–dataset combinations and in every dataset and every model evaluated (mean ΔAUC =  − 0.028). Because the combinations are crossed rather than independent, clustering-aware analyses (by dataset, by model, and a crossed mixed-effects model, 95% CI − 0.062 to + 0.006) gave the same − 0.028 mean degradation—directionally consistent across every dataset and every model, though with an imprecise magnitude; the degradation worsened monotonically as regularization weakened and was robust to classifier choice (logistic regression and MLP), numerical precision (FP16 vs. FP32), and image resolution. A variance-retention sweep located an empirical operating region of 90–95% beyond which degradation re-emerged. Whitening also degraded retrieval (ΔmAP =  − 0.149) and few-shot learning (Δ5-shot =  − 0.224), indicating that anisotropic geometry encodes task-relevant structure across all three tasks. Notably, a near-isotropic model (PLIP, AvgCosSim ≈ 0.003) degraded comparably to the most anisotropic models, indicating that the degradation reflects the variance structure of the eigenspectrum rather than anisotropy itself. We propose spectral-aware standardization (SAS), a gentle spectral transform that improves classification (ΔAUC =  + 0.005, 23/36; validated leave-one-dataset-out and on two held-out pathology models, UNI and Virchow, 12/12) and, unlike whitening, does so without large retrieval or few-shot penalties—a Pareto improvement over whitening across the three tasks. The gains over simple <i>z</i>-score standardization are nonetheless marginal, and for distance-based use, plain standardization remains the recommended default.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Embedding Anisotropy in Medical Image Foundation Models: Post Hoc Whitening Degradation and Spectral-Aware Standardization

  • 期刊论文

  • 发表日期：

    2026年7月7日

查看全文
  • Clemente García-Hidalgo

摘要

Foundation models produce high-dimensional embedding representations whose geometric properties fundamentally determine downstream task effectiveness. Embedding anisotropy—the concentration of representations in a narrow cone—has been documented in natural language processing but not systematically studied in vision foundation models. We characterize anisotropy in embeddings from eight architecturally diverse models (ResNet50, DINOv2-S/L, BiomedCLIP, PLIP, SigLIP, UNI, and Virchow) across six MedMNIST benchmarks and two native-resolution clinical datasets (NCT-CRC-HE-100K and ISIC 2019; N > 110,000 images total). All models exhibited substantial anisotropy: effective dimensionality ranged from 0.7 to 7.1% of the nominal embedding dimension. Post hoc whitening (PCA/ZCA) reduced classification AUC in 34 of 36 model–dataset combinations and in every dataset and every model evaluated (mean ΔAUC =  − 0.028). Because the combinations are crossed rather than independent, clustering-aware analyses (by dataset, by model, and a crossed mixed-effects model, 95% CI − 0.062 to + 0.006) gave the same − 0.028 mean degradation—directionally consistent across every dataset and every model, though with an imprecise magnitude; the degradation worsened monotonically as regularization weakened and was robust to classifier choice (logistic regression and MLP), numerical precision (FP16 vs. FP32), and image resolution. A variance-retention sweep located an empirical operating region of 90–95% beyond which degradation re-emerged. Whitening also degraded retrieval (ΔmAP =  − 0.149) and few-shot learning (Δ5-shot =  − 0.224), indicating that anisotropic geometry encodes task-relevant structure across all three tasks. Notably, a near-isotropic model (PLIP, AvgCosSim ≈ 0.003) degraded comparably to the most anisotropic models, indicating that the degradation reflects the variance structure of the eigenspectrum rather than anisotropy itself. We propose spectral-aware standardization (SAS), a gentle spectral transform that improves classification (ΔAUC =  + 0.005, 23/36; validated leave-one-dataset-out and on two held-out pathology models, UNI and Virchow, 12/12) and, unlike whitening, does so without large retrieval or few-shot penalties—a Pareto improvement over whitening across the three tasks. The gains over simple z-score standardization are nonetheless marginal, and for distance-based use, plain standardization remains the recommended default.