Scholarlink

Large language models (LLMs) are powerful but resource-intensive, limiting accessibility. HITgram addresses this gap by offering a lightweight platform for n-gram model experimentation, ideal for resource-constrained environments. It supports unigrams to 4-grams and incorporates features like context-sensitive weighting, Laplace smoothing, and dynamic corpus management to enhance prediction accuracy, even for unseen word sequences. Experiments demonstrate HITgram’s efficiency, achieving 50,000 tokens/second and generating 2-grams from a 320 MB corpus in 62 s. HITgram scales efficiently, constructing 4-grams from a 1 GB file in under 298 s on an 8 GB RAM system. Planned enhancements include multilingual support, advanced smoothing, parallel processing, and model saving, further broadening its utility.

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

HITgram: A Platform for Experimenting with n-Gram Language Models

  • 会议论文

  • 发表日期：

    2025年

查看全文
  • Shibaranjani Dasgupta,
  • Chandan Maity,
  • Somdip Mukherjee,
  • Rohan Singh,
  • Diptendu Dutta,
  • Debasish Jana

摘要

Large language models (LLMs) are powerful but resource-intensive, limiting accessibility. HITgram addresses this gap by offering a lightweight platform for n-gram model experimentation, ideal for resource-constrained environments. It supports unigrams to 4-grams and incorporates features like context-sensitive weighting, Laplace smoothing, and dynamic corpus management to enhance prediction accuracy, even for unseen word sequences. Experiments demonstrate HITgram’s efficiency, achieving 50,000 tokens/second and generating 2-grams from a 320 MB corpus in 62 s. HITgram scales efficiently, constructing 4-grams from a 1 GB file in under 298 s on an 8 GB RAM system. Planned enhancements include multilingual support, advanced smoothing, parallel processing, and model saving, further broadening its utility.