Scholarlink

We study serving retrieval models, particularly late interaction retrievers like ColBERT, to many concurrent users at once and under a small budget, in which the index may not fit in memory. We present ColBERT-serve, a serving system that applies a memory-mapping strategy to the ColBERT index, reducing RAM usage by 90% and permitting its deployment on cheap servers, and incorporates a multi-stage architecture with hybrid scoring, reducing ColBERT’s query latency and supporting many concurrent queries in parallel.

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

ColBERT-Serve: Efficient Multi-stage Memory-Mapped Scoring

  • 会议论文

  • 发表日期：

    2025年

查看全文
  • Kaili Huang,
  • Thejas Venkatesh,
  • Uma Dingankar,
  • Antonio Mallia,
  • Daniel Campos,
  • Jian Jiao,
  • Christopher Potts,
  • Matei Zaharia,
  • Kwabena Boahen,
  • Omar Khattab,
  • Saarthak Sarup,
  • Keshav Santhanam

摘要

We study serving retrieval models, particularly late interaction retrievers like ColBERT, to many concurrent users at once and under a small budget, in which the index may not fit in memory. We present ColBERT-serve, a serving system that applies a memory-mapping strategy to the ColBERT index, reducing RAM usage by 90% and permitting its deployment on cheap servers, and incorporates a multi-stage architecture with hybrid scoring, reducing ColBERT’s query latency and supporting many concurrent queries in parallel.