The spoofing countermeasure enhances the system’s ability to defend against various attacks, ensuring its security and reliability. However, attackers using speakers to replay fake speech to deceive automatic speaker verification (ASV) systems have become more prevalent. Therefore, this paper proposes an audio replay spoof attack detection by a joint F-ratio adaptive filter bank of the silent segment and attention-enhanced ResNeXt-50 network. First, the benefits of feature extraction in silent segments are theoretically and practically analyzed. Second, an adaptive filter bank based on F-ratio can provide important information about spectrum variability, and extract key information in the audio silent segments by dynamically assigning filters in eight spectrum intervals. Finally, an attention-enhanced ResNeXt-50 network introduces cardinality concepts, group convolution, and embedding attention, improving the processing performance. The experimental results show that our network achieves the lowest EER of 1.57% and the third lowest min t-DCF of 0.1709 compared to the SOTA methods.