Driver fatigue is among the major causes of accidents, and hence real-time detection is needed. Here, we propose a multimodal detection scheme combining YOLOv11 for visual examination, transformer-based temporal modeling, and physiological monitoring. Combining the eye closure patterns, yawning, heart rate variability, and vehicle telemetry data enhances accuracy and reliability. Evaluated on the NITYMED dataset (EEG, ECG, and face video data), our approach outperforms state-of-the-art methods by 14.6% accuracy and lowers inference latency from 50ms to 15ms. Federated learning provides privacy-preserving model updates, thus making it suitable for use across drivers. Our approach outperforms vision-based models and CNN-LSTM with 9.3% fewer false positives. The system is optimized for real-time edge deployment with low computational cost. Scalability, environmental robustness, and energy-efficient deployment will be treated in future work.