This paper addresses the challenge of defending neural networks against adversarial attacks and environmental noise by proposing a cross-modal architecture that combines image and audio data streams. With the MNIST and Audio MNIST datasets, the model integrates specialized Convolutional Recurrent Neural Network (CRNN) encoders for each modality, followed by a late fusion layer that synthesizes features into a unified, resilient representation. This fusion mechanism exploits the inherent consistency between modalities, reducing the impact of adversarial perturbations targeting a single data stream. Preprocessing steps, including normalization, Gaussian noise injection, and adversarial example generation through FGSM and PGD, are applied to simulate diverse attack scenarios. The adversarial training loop, designed to expose the model to clean and adversarial samples, further strengthens its robustness by minimizing the adversarial success rate (ASR) while preserving high accuracy. Evaluation metrics like F1-score, precision, recall, and accuracy confirm the effectiveness of this defense strategy. Our results highlight a marked improvement in performance over unimodal approaches, with a substantial reduction in ASR and enhanced resilience in noisy environments. By uniting cross-modal consistency checks and adversarial training, this approach offers a promising defense mechanism for real-world applications where security and reliability are paramount.