TokenPacker: Efficient Visual Projector for Multimodal LLM
摘要
In multimodal large language models (MLLMs), the visual projector is a crucial component that connects the visual encoder with the large language model (LLM). Most current MLLMs adopt a simple multi-layer perceptron (MLP) to preserve visual contexts via direct transformation. However, this approach tends to generate redundant visual tokens, particularly when processing high-resolution images, ultimately reducing the efficiency of MLLMs. Recent efforts to address this issue have employed resamplers or abstractors to reduce token quantity. Unfortunately, these methods often fail to capture finer details, thereby limiting the model’s visual reasoning capabilities. In this work, we introduce TokenPacker, a novel visual projector designed to generate condensed visual tokens through a coarse-to-fine scheme. Initially, we interpolate the visual features into a low-resolution point query that provides an overall visual representation. We then integrate high-resolution, multi-level regional cues using a region-to-point injection module, which enriches the point query with local context. This enhancement effectively transforms the initial query into a more detailed representation suitable for LLM reasoning. Furthermore, we propose a dynamic image slicing scheme to efficiently handle high-resolution images with TokenPacker. Extensive experiments demonstrate that TokenPacker can compress the visual tokens by 75%