In a model inversion attack, an adversary tries to reconstruct private training data through iterative inference of a neural network model. To ensure confidentiality of the training data, protection against model inversion attacks is crucial. However, existing defense techniques primarily require modifications to the trained model architecture or even retraining, which limits their applicability to deployed models. In addition, most of them become ineffective against label-only attacks, which require minimal information to succeed. This paper proposes an improved defense mechanism that filters out inference requests from malicious attackers. It does not alter existing model architectures and works against various types of attacks. Our experimental results show that our approach is highly effective with mean defense accuracy scores of 90.00% and 95.83% on two datasets, respectively. Therefore, the proposed approach has been proven to be successful in defending against contemporary model inversion attacks, thus achieving the objectives of this study.