Convolutional Neural Networks (CNNs) are neural network (NN) models that are predominantly being used in the domain of Computer vision (CV). The popularity of these models can be attributed to their exceptional predictive and automatic feature extraction capabilities. CNNs, like other neural network models, are trained using the algorithm of back-propagation, which is an iterative process. The back-propagation algorithm suffers from certain drawbacks, like slow convergence, being hypersensitive to the learning rate, converging in a local minimum and so on. This iterative training using back-propagation affects the accuracy of the model and is also computationally heavy. We propose to train the classifier part of the CNN in a non-iterative manner. In the proposed method, relevant features are automatically extracted from the raw data and the Gram Schmidt method is used to decompose this feature matrix and learn the weights for the classifier part of the model. Our method has demonstrated better performance in terms of accuracy metric compared to other state-of-the-art methods. The proposed method has been tested on two benchmark data sets, MNIST and CIFAR-10.