Large language models (LLMs) have demonstrated impressive capabilities in various downstream tasks, particularly in zero-shot scenarios. However, their performance in complex Information Extraction (IE) tasks, especially those involving long tail knowledge, remains limited. In this work, we propose an Open-World Retrieval-Augmented Generation (RAG) framework specifically designed for zero-shot IE with LLMs. By dynamically incorporating external knowledge from open-world sources, our approach addresses the challenges of rare entities and complex relationships, reducing model confusion and enhancing extraction accuracy. Experiments on three IE tasks show that Open-World RAG significantly boosts performance across model sizes, highlighting its potential as a scalable zero-shot IE solution.