Road transpiration system highly depends on the road conditions and poor road conditions are responsible for accidents and vehicle safety. In the era of automation and self-driving car, good road conditions are necessitated for the effective transportation and user safety. To execute timely road maintenance, efficient methods to detect the poor road conditions are required. In this direction, this paper provides an Xception model-based method for the classification of road images. 1220 self-created road images are used in this experiment. These images are categorized into four parts named normal roads, large pothole, small pothole, and cracks. The Xception model is implemented on this dataset, and four different data division strategies were considered to analyze the model behavior on unseen data and different amounts of training and test data. According to the experiment performed, the Xception model provided the highest accuracy of 97.83%. To evaluate the performance of the model, precision, recall, and F1-score-based analysis are performed. The performance of the Xception model is compared to the InceptionResNet and CNN models using different data division strategies and epoch counts, and it was observed that the InceptionResNet and CNN models attained the highest accuracy of 97.12% and 92.05%, respectively, on the same dataset.