Parameter-efficient fine-tuning (PEFT) plays a crucial role in the rapid development and application of large models, among which Low-Rank Adaptation (LoRA) is a mainstream method that fine-tunes large language models (LLMs) through training additional adapters. Recent studies adopt an unmerged strategy for multiple LoRA fine-tuned model inference to improve inference efficiency. However, this strategy requires synchronized inference, thus leading to additional latency. Therefore, this study proposes Fusion Adapter LoRA Inference (FALI), a novel method for collaboratively accelerating inference across multiple LoRA fine-tuned models. The FALI fusion adapter for large-scale matrix multiplication fully utilizes general matrix multiplication (GEMM) to improve overall inference efficiency in parallel. Experimental results demonstrate the effectiveness of FALI, and its potential future extensions, including system-level optimization and distributed implementation, are also discussed.