Scholarlink

Background <p>Single-cell foundation models such as scGPT and Geneformer learn rich representations of gene expression programs, but whether these representations encode gene regulatory relationships beyond expression-level confounds remains unclear. Attention patterns in these models have been shown to capture co-expression rather than direct regulation, leaving open the question of whether deeper representations—particularly the residual stream—contain genuine regulatory information.</p> Results <p>We systematically investigated residual-stream geometry in scGPT and Geneformer across four tissue contexts from the Tabula Sapiens atlas, evaluating whether geometric proximity between gene vectors provides incremental predictive value for curated TRRUST transcription factor–target edges beyond expression confounds. Under repeated stratified cross-validation, geometric features provided significant incremental signal in kidney and immune settings, validated by label-permutation and geometry-shuffle null controls; centered-cosine similarity, PCA projection and multi-layer bundling recovered comparable signal in lung tissues, and the multi-layer bundle improved every domain (kidney ΔAUROC&#xa0;=&#xa0;+&#xa0;0.122, immune +&#xa0;0.042, lung +&#xa0;0.028, external lung +&#xa0;0.027; geometry-augmented AUROC 0.60–0.69). The effect was fully robust to leave-TF-out and leave-target-out cross-validation and to harder degree- and expression-matched negative edges, but under the stricter leave-both-out split—no transcription factor and no target shared between folds—it collapsed to near-zero (ΔAUROC at most +&#xa0;0.003, and not statistically significant in kidney or immune), marking the ceiling of out-of-entity generalization. With a comparable per-layer residual-stream extraction applied to both models, the apparent Geneformer advantage mostly disappeared (small residual gaps remained in three of four domains), indicating it largely reflected representation-construction choices rather than a substantial architectural difference. Asymmetric geometric features predicted regulatory edge orientation (AUROC 0.80–0.90), and the geometric signal added incremental value on top of expression-based gene regulatory network (GRN) inference (GENIE3, co-expression).</p> Conclusion <p>Foundation model residual streams carry incremental, regulatory-relevant geometric signal that is distributed across layers and that complements expression-based GRN inference for retrospective edge prioritization. The signal is statistical enrichment rather than a stand-alone regulatory classifier: absolute performance is modest and out-of-entity generalization is limited, so its practical role is as an orthogonal evidence channel for edge re-ranking and hypothesis prioritization in multi-evidence frameworks.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Residual-stream geometry of single-cell foundation models carries incremental gene-regulatory signal across tissues

  • 期刊论文

  • 出版模式：

    开放获取

  • 发表日期：

    2026年7月24日

查看全文
  • Ihor Kendiukhov

摘要

Background

Single-cell foundation models such as scGPT and Geneformer learn rich representations of gene expression programs, but whether these representations encode gene regulatory relationships beyond expression-level confounds remains unclear. Attention patterns in these models have been shown to capture co-expression rather than direct regulation, leaving open the question of whether deeper representations—particularly the residual stream—contain genuine regulatory information.

Results

We systematically investigated residual-stream geometry in scGPT and Geneformer across four tissue contexts from the Tabula Sapiens atlas, evaluating whether geometric proximity between gene vectors provides incremental predictive value for curated TRRUST transcription factor–target edges beyond expression confounds. Under repeated stratified cross-validation, geometric features provided significant incremental signal in kidney and immune settings, validated by label-permutation and geometry-shuffle null controls; centered-cosine similarity, PCA projection and multi-layer bundling recovered comparable signal in lung tissues, and the multi-layer bundle improved every domain (kidney ΔAUROC = + 0.122, immune + 0.042, lung + 0.028, external lung + 0.027; geometry-augmented AUROC 0.60–0.69). The effect was fully robust to leave-TF-out and leave-target-out cross-validation and to harder degree- and expression-matched negative edges, but under the stricter leave-both-out split—no transcription factor and no target shared between folds—it collapsed to near-zero (ΔAUROC at most + 0.003, and not statistically significant in kidney or immune), marking the ceiling of out-of-entity generalization. With a comparable per-layer residual-stream extraction applied to both models, the apparent Geneformer advantage mostly disappeared (small residual gaps remained in three of four domains), indicating it largely reflected representation-construction choices rather than a substantial architectural difference. Asymmetric geometric features predicted regulatory edge orientation (AUROC 0.80–0.90), and the geometric signal added incremental value on top of expression-based gene regulatory network (GRN) inference (GENIE3, co-expression).

Conclusion

Foundation model residual streams carry incremental, regulatory-relevant geometric signal that is distributed across layers and that complements expression-based GRN inference for retrospective edge prioritization. The signal is statistical enrichment rather than a stand-alone regulatory classifier: absolute performance is modest and out-of-entity generalization is limited, so its practical role is as an orthogonal evidence channel for edge re-ranking and hypothesis prioritization in multi-evidence frameworks.