<p>Convolutional neural network models have been developed over the past few decades demonstrating remarkable performance in image classification tasks for various applications in industries. The potential to produce higher accuracy has resulted in deep convolutional neural networks widely used in research where the presence of complex computations is the secret of success. Larger complex computations cause an increased inference time for a single classification, which eventually leads to overall latency of vision-based systems. This research work focuses on minimizing the inference time of a convolutional neural network while achieving better classification accuracy by cascading modified convolutional neural network models. Learnable layers of a CNN, such as convolutional and fully connected layers, are involved in heavy complex computations that account for most of the inference time. Modified architecture with optimized computations while maintaining generalization capability leads to minimizing latency. The cascaded neural network model trained on a lesser number of output classes offered better classification accuracies and reduced inference time compared to standard convolutional neural network models such as Alex net. The performance of the cascaded neural network model was tested for classification accuracy and execution time for a wide range of test data presented, that proposed cascaded network inference speed was enhanced by 6% while maintaining the classification accuracy as the AlexNet for tests conducted on 200 object classes.</p>