Sport is a universal language that captivates audiences worldwide with its combination of strategy, athleticism, and fierce competition. Of all the sports, football is one of the most popular worldwide, renowned for its mass appeal and cultural significance. Predicting the direction of penalty kicks or shots for some particular players with advanced technology solutions is an exciting opportunity for research in sports video analytics. This paper introduces a novel penalty image dataset compiled from diverse penalty kick scenarios. Then we propose a convolutional neural network (CNN)-based model utilizing the concatenation of InceptionV3 and ResNet101V2, pre-trained models. Our proposed model gives an accuracy of 57.89%, which may provide useful information to goalkeepers and coaches.