A large mass of the world’s population depends on agriculture, and as the population increases, the demand for food also increases. Weeds in the agriculture field hampered the productivity and quality of the agriculture. This issue can be overcomed by deep learning (DL)-based weed detection and classification approach, which is efficient, automated, cost-effective, and environment friendly. This work primarily aims to perform a comparative analysis of several pre-trained convolutional neural network (CNN) models to fill the research gap regarding the classification of weed and non-weed in paddy crops. The image dataset has been collected from paddy fields near Baripada, Mayurbhanj, Odisha, and implemented in this research. Four pre-trained DL models, such as VGG16, DenseNet121, ResNet101, and MobileNetV2, have been investigated for the classification. In this study, we have obtained that MobileNetV2 is the most efficient model achieving a success rate of 98.36% in classifying weed and non-weed plants in the paddy crop field.