<p>Recently, numerous studies have been carried out to comprehend the mechanics of deepfakes, and various deep learning (DL) approaches have been proposed to identify deepfake videos or images. They can accurately detect one of three types of attacks: digital manipulation, adversarial, or physical spoofs. However, they do not perform better when focused on all three attacks. In order to overcome this shortcoming, a novel, and optimal DL-based attack detection framework is proposed, which can automatically classify 20 types of coherent attacks based on three types. The proposed system mainly involves image preprocessing, feature extraction, and classification. Initially, the images from the dataset are preprocessed using You Look Only Once version 5 (YOLOV5). Next, the feature learning scheme initially captures the images' texture features by employing the Butterfly Optimized Gabor Filter with Local Binary Pattern (BOGFLBP). Then, a Deformable Convolution-Based Xception Network (DCXCN) is applied for spatial feature extraction. Then, the features from BOGFLBP and DCXCN are fused and inputted to the Bidirectional Long Short-Term Memory (BLSTM) for temporal feature extraction. The hyperparameters of the learning network are optimally chosen using the Double exponential distribution-based Chicken Swarm Optimization (DCSO) to improve its detection performance. Finally, the dense layers classify the different attacks on collected videos. The outcomes represent the superiority of the proposed system for joint attack detection compared to existing models.</p>