Document Visual Question Answering (DocVQA) is critical for automating the comprehension of visually rich documents in domains such as finance, healthcare, education, and law. However, existing retrieval-augmented generation (RAG) methods, such as TextRAG and VisRAG, face significant challenges in handling complex document layouts and large-scale document corpora. TextRAG, reliant on OCR-based text extraction, often fails to capture essential visual structures like tables and diagrams, resulting in incomplete semantic representations. VisRAG, while incorporating visual cues, suffers from performance degradation due to accumulated visual noise when increasing document length. To address these challenges, we propose a novel two-stage framework comprising dual-stream adaptive retrieval and hierarchical agent collaboration. The retrieval module integrates textual and visual retrievers, guided by a Wasserstein-based Compact Adaptive Breakpoint Selection (WCABS) algorithm, which dynamically optimizes the selection of top-k evidence pages to minimize noise across multi-page documents. The reasoning module employs a hierarchical agent system, where a Selector Agent performs distributed evidence extraction and an Integrator Agent aggregates the extracted information to generate the final answer, ensuring robust and structured reasoning. Experiments on the ViDoSeek dataset demonstrate the effectiveness of our approach, showcasing robust multimodal retrieval and structured reasoning capabilities for complex document queries.