<p>Deep learning has been increasingly applied to impact-echo (IE) signals for automatic detection of internal defects in concrete structures. Existing studies typically convert IE signals into a single time–frequency map and focus on optimizing convolutional neural network (CNN) architectures. However, it remains unclear whether the classification performance is mainly determined by the CNN structure or by the time–frequency representation of the signal. This study systematically compares four representative time–frequency representations—STFT, bump, morse, and amor wavelet transforms—combined with four lightweight CNN models under identical datasets, training settings, and noise conditions. IE signals collected from a concrete slab containing defects at different depths were converted into time–frequency images for classification. The results show that the choice of time–frequency representation has a more significant influence on classification performance than the CNN architecture. Morse and amor wavelets consistently provide clearer feature patterns and stronger noise robustness than Bump and STFT. Among all combinations, the amor–MobileNet model achieves the best performance with an accuracy exceeding 99%. The findings indicate that appropriate signal representation plays a dominant role in CNN-based IE signal classification and provide practical guidance for deep learning–based nondestructive evaluation of concrete structures.</p>