Static analysis tools are widely used in software engineering practice, but they suffer from a high false positive rate. In this paper, we propose an automated approach to mitigate false alarms based on large language models (LLMs). Our approach relies on the collaboration of three agents. The source code is sliced and incrementally provided to the agents on demand. By leveraging few-shot learning and beam search with self-reflection, our method effectively filters out false alarms. On an artificial test suite, it eliminates 91.9% of false alarms reported by traditional static analysis tools. The accuracy of bug reports (measured by the F1 score) is significantly improved from 0.575 to 0.913. Experimental results on real-world projects from GitHub show that false alarms are reduced by an average of 57.2% and up to 96.6%. Our approach is not a replacement for traditional static analysis tools but rather an enhancement. Therefore, it can be easily integrated with any static analysis tool to provide more accurate and efficient analysis reports.