Lowering Costs and Increasing Benefits Through the Ensemble of LLMs and Machine Learning Models

Due to the remarkable capabilities of generative AI technologies, pre-trained large language models (LLMs) have been widely used. However, the high computational and memory requirements limit the widespread adoption of LLMs. It is a challenging issue that how to reduce the cost of using LLMs while maintaining their performance as much as possible. In this paper, we explore to integrate LLMs with small-scale machine learning models through ensemble learning. We propose a collaborative training and testing framework, which mainly includes a LLM, a small machine learning model and a task allocator. In this framework, we can train the small machine learning model, and further train the task allocator using the data generated by both the LLM and the small machine learning model. The task allocator is trained to learn how to assign tasks to the LLM or the small machine learning model. We design the training method of task allocator and give its implementation algorithm. To validate our method, we conducted experiments on two tasks, sentiment classification and text summarization. The results demonstrate that our approach can improve the performance while reducing the overall usage costs.