This work presents a comparative study of advanced methods focused on exploring strategies to improve the classification of two commercially important Mediterranean fish species, S. aurata and D. labrax, as wild or farmed. This task is crucial for environmental conservation and aquaculture management, as it supports responsible aquaculture practices and maintains consumer confidence by verifying the origin and authenticity of fish products. Approaches, such as CNNs and ViTs, often rely on task-specific training, limiting their adaptability across domains. In this work, we compare the potential of CLIP for multimodal fish classification. By combining CLIP’s pretrained architecture with a lightweight linear classifier and incorporating real-world textual descriptions from domain experts, we achieve high classification accuracy with minimal task-specific training. The results demonstrate that CLIP, even with a simple linear probe, surpasses convolutional models in accuracy, generalization and adaptability, showcasing its potential for niche classification tasks and broader applications.