Deep neural networks (DNNs) have become prevalent and being used across various fields. Meanwhile, their extensive use has been raising some major security concerns. DNNs can be fooled by an adversary as a small intelligent change in input would cause change of output label. Existing methodologies require model retraining on a poisoned dataset or inserting additional multilayer perceptron (MLPs), which involves additional computation and time constraints. This work presents a backdoor attack, which generates a small patch to misclassify the prediction, if added to the image. Interestingly, the patch does not affect the physical appearance of the image. The patch is generated by determining influential features through sensitivity analysis. Subsequently, negative-contributing features are generated as the intended patch using Intersection over union (IoU). The most interesting part of our proposed technique is that it does not require the model re-training or any alterations to the model. Experiments on three different types of datasets (MNIST, CIFAR-10, and GTSRB) demonstrate the effectiveness of the attack. It is observed that our proposed method achieves a higher attack success rate around 50–70%, without compromising the test accuracy for clean input samples.