Phishing is a widely employed malicious tactic where attackers use emails to impersonate trusted individuals. Although phishing has been a well-known cybercrime strategy for decades, its prevalence has grown in recent years, particularly during the COVID-19 pandemic. In this paper, we investigate the effectiveness of using large language models (LLMs) to detect phishing attacks. Our approach utilizes LLMs that are fine-tuned for phishing classification. Furthermore, we develop a range of methods that utilize an ensemble of fine-tuned models that leverage the natural diversity among fine-tuned models. Additionally, we propose a two-stage approach that uses an ensemble of low-cost LLMs along with an ensemble of high-cost, highly accurate LLMs. The methods are evaluated across a wide range of LLMs and on a data set comprised of 5,882 emails equally distributed over phishing and benign. This dataset was collected from two well-known data sets: The Enron corpus and the most recent emails from the Nazario phishing corpus.