Existing methods for fine-grained cat–dog breed and individual recognition often suffer from reduced accuracy, class imbalance, and limited generalization due to subtle inter-breed differences, high intra-breed variability, and complex factors such as pose, illumination, and occlusion. To address these challenges, we propose a lightweight framework based on a Progressive Co-Attention Network (PCA-Net) that enables inter-channel feature interactions for robust shared representations. A label-aware smoothing strategy alleviates class imbalance, while contrastive learning enhances feature discriminability and generalization. Using EfficientNet as the backbone, the model size is maintained at 44.09 MB. Experiments on ImageNet Dogs, Pets, Synthesis, and ThuDogs demonstrate consistent accuracy above 90%, with a maximum of 97.50%, highlighting the method’s effectiveness in balancing accuracy and computational efficiency.