Large language models (LLMs) have shown strong abilities in understanding and generating natural language, offering new opportunities for clinical text analysis. Most prior studies, however, rely on proprietary systems, raising concerns about data privacy and accessibility in healthcare. Open-source LLMs offer transparent, locally deployable, and privacy-preserving alternatives, yet their performance in low-resource languages and zero-shot medical information extraction remains underexplored. We evaluated modern open-source generative LLMs for extracting clinically relevant information from Dutch medical reports using the DRAGON 2024 benchmark. To enable this, we developed llm-extractinator [1], an open-source framework that automates structured information extraction via schema definition, dynamic prompt construction, local inference, and automatic validation. Nine multilingual models were evaluated on 28 DRAGON tasks spanning classification, regression, and named entity recognition in a strict zero-shot setup. Performance was measured using task-specific metrics (AUC, Cohen’s K, RSMAPE, F1) and aggregated into the DRAGON utility score. Llama-3.3-70B achieved the highest score (SDRAGON = 0.760), followed by Phi-4-14B (0.751), Qwen-2.5-14B (0.748), and DeepSeek-R1-14B (0.744). These models matched or exceeded a fine-tuned RoBERTa baseline on 17 of 28 tasks. Translating Dutch input to English reduced performance (ΔS = −0.11 to −0.25), emphasizing the importance of native-language inference. Our results show that open-source generative LLMs can achieve competitive performance without fine-tuning, providing a practical and privacy-preserving solution for clinical information extraction in resource-constrained settings. The llm-extractinator framework facilitates reproducible benchmarking and lowers the barrier for applying LLMs in local medical research environments.