Controlling drone swarms in Human-in-the-loop systems can be a complex task, especially for swarms consisting of multiple drones. Cognitive complexity is (in addition to other factors) heavily impacted by the control modality. Although multiple control strategies have been proposed already, many do not scale well with larger numbers of drones, require special training, or are restricted regarding the available instructions. In this paper, a novel integrated Large Language Model (LLM) and Augmented Reality (AR)-based control strategy for drone swarms is presented to tackle the before-mentioned issues. The core idea of the proposed control is to offer users a voice-command interface to issue natural-language commands. These are then analyzed by a LLM which generates target coordinates for the drones and an auditory response which is played to the user. Position, target and state information can be observed using an AR-headset which also contains the hardware for the voice interaction. To evaluate our approach, a user study with 17 participants was conducted with results indicating high usability and a low workload imposed on the participants. The approach was also compared to a traditional, manual controller-based modality which utilized the same AR-based visualizations. The evaluation results indicate that the usability and workload of the novel LLM modality and the manual controller-based modality were fairly similar. Although the manual controller-based approach showed better time efficiency, it was perceived as imprecise by many participants, while the LLM control was often perceived as precise and well-suited to carry out complex tasks.