Scholarlink

We present Prompting the Mind (PTM), an extended EEG-to-text translation framework that combines large language models (LLMs) with multimodal alignment to decode human brain signals into natural language. Our system follows a multi-stage pipeline: an EEG encoder first transforms raw neural activity into discriminative embeddings; these are then mapped into a shared vision-language semantic space using CLIP-based cross-modal alignment. Finally, a general-purpose base LLM, DeepSeek-7B-Base, generates descriptive text conditioned on the EEG-derived representations through structured prompting. We evaluate the framework on a publicly available EEG-image dataset, comparing its performance with chance-level and alignment-only baselines as well as an instruction-tuned LLM (Mistral-7B). Results on BLEU, METEOR, ROUGE-L, and BERTScore show that while instruction-tuned models yield higher token overlap, our prompt-conditioned base LLM produces shorter, more semantically faithful outputs that better align with the original brain signals. Qualitative examples highlight this trade-off and the practical value of structured prompting for non-invasive neural decoding. All code, prompt templates, and configuration files are shared ( https://github.com/Sadi-Mahmud-Shurid/PTM ) to promote reproducibility and future extensions of open-weight frameworks for brain-to-text communication.

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Prompting the Mind: EEG-to-Text Translation with Multimodal LLMs and Semantic Control

  • 会议论文

  • 发表日期：

    2026年

查看全文
  • Mohammed Salah Al-Radhi,
  • Sadi Mahmud Shurid,
  • Géza Németh

摘要

We present Prompting the Mind (PTM), an extended EEG-to-text translation framework that combines large language models (LLMs) with multimodal alignment to decode human brain signals into natural language. Our system follows a multi-stage pipeline: an EEG encoder first transforms raw neural activity into discriminative embeddings; these are then mapped into a shared vision-language semantic space using CLIP-based cross-modal alignment. Finally, a general-purpose base LLM, DeepSeek-7B-Base, generates descriptive text conditioned on the EEG-derived representations through structured prompting. We evaluate the framework on a publicly available EEG-image dataset, comparing its performance with chance-level and alignment-only baselines as well as an instruction-tuned LLM (Mistral-7B). Results on BLEU, METEOR, ROUGE-L, and BERTScore show that while instruction-tuned models yield higher token overlap, our prompt-conditioned base LLM produces shorter, more semantically faithful outputs that better align with the original brain signals. Qualitative examples highlight this trade-off and the practical value of structured prompting for non-invasive neural decoding. All code, prompt templates, and configuration files are shared ( https://github.com/Sadi-Mahmud-Shurid/PTM ) to promote reproducibility and future extensions of open-weight frameworks for brain-to-text communication.