<p>Dialogue discourse parsing is a fundamental task in natural language understanding. It aims to capture the relationships between utterances in a dialogue, facilitating a deeper understanding of dialogue structures and semantics, especially in long and complex dialogues. Existing research often develops separate dialogue discourse parsers for text-only and multimodal scenarios, largely due to the scarcity of parallel multimodal annotated datasets. This separation limits the ability to fully utilize diverse data with different modalities and poses challenges for real-world artificial intelligence applications. To address the limitation, we propose a unified dialogue discourse parsing framework that bridges text-only and multimodal parsing within a single model. We first develop a basic text-only parser, pre-trained on textual datasets. Then, we extend it to multimodal scenarios by adding additional multimodal encoders and fusion modules, while freezing the parameters learned during the text-only stage. We conduct extensive experiments on three datasets, covering both text-only and multimodal dialogues. Experimental results show that our approach achieves significant average improvements over several existing benchmarks. This demonstrates the generalizability and effectiveness of our framework for dialogue discourse parsing across different modalities.</p>