Scholarlink

Large Language Models (LLMs) typically track the order of tokens using positional encoding, which causes the following problems: positional bias, where the model is influenced by an ordering within the prompt, and a fixed context window, as models struggle to generalize to positions beyond those encountered during training. To address these limitations, we developed a novel method called set encoding. This method allows multiple pieces of text to be encoded in the same position, thereby eliminating positional bias entirely. Another promising use case for set encoding is to increase the size of the input an LLM can handle. Our experiments demonstrate that set encoding allows an LLM to solve tasks with far more tokens than without set encoding. To our knowledge, set encoding is the first technique to effectively extend an LLM’s context window without requiring any additional training. (This is an extended abstract of a paper that has been presented at the 63rd Annual Meeting of the Association for Computational Linguistics (ACL) 2025 in Vienna (AT)).

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Positional Overload: Positional Debiasing and Context Window Extension for Large Language Models Using Set Encoding

  • 会议论文

  • 发表日期：

    2026年

查看全文
  • Lukas Kinder,
  • Lukas Edman,
  • Alexander Fraser,
  • Tobias Käfer

摘要

Large Language Models (LLMs) typically track the order of tokens using positional encoding, which causes the following problems: positional bias, where the model is influenced by an ordering within the prompt, and a fixed context window, as models struggle to generalize to positions beyond those encountered during training. To address these limitations, we developed a novel method called set encoding. This method allows multiple pieces of text to be encoded in the same position, thereby eliminating positional bias entirely. Another promising use case for set encoding is to increase the size of the input an LLM can handle. Our experiments demonstrate that set encoding allows an LLM to solve tasks with far more tokens than without set encoding. To our knowledge, set encoding is the first technique to effectively extend an LLM’s context window without requiring any additional training. (This is an extended abstract of a paper that has been presented at the 63rd Annual Meeting of the Association for Computational Linguistics (ACL) 2025 in Vienna (AT)).