Marine plastic pollution poses a critical threat to aquatic ecosystems, biodiversity, and global sustainability. Existing proposals on the detection and mitigation of plastic waste in oceans are limited by fragmented data sources, lack of real-time capabilities, and low scalability. This work proposes a novel deep learning framework that integrates multi-modal data sources and crowdsourced reports to detect, classify, and predict the movement of plastic debris in marine environments. The proposed OCEAN-Net framework utilizes a multi-branch neural network comprising Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs), and transformers. Novel features of this framework include the incorporation of microplastic detection, self-supervised learning for data-efficient pre-training, and explainable AI (XAI) techniques to enhance model interpretability. The proposed approach was validated using diverse datasets, achieving significant improvements in detection accuracy up to 95% and drift prediction reliability 89% over baseline methods. The model is also aimed at identifying high-density plastic zones, aiding targeted cleanup operations and informing policy decisions. This work contributes to the broader goal of sustainable oceans by offering an innovative, scalable, and actionable solution to marine plastic pollution.