Deep classification models have been criticized for showing noise sensitivity, bias toward background, and lack of interpretability. In this paper, we expand the idea of using local post hoc interpretation to improve classification accuracy under the assumption of existing object localization masks. Our classifiers are trained to align their interpretation maps with these objects, and we refer to such models as SelfExplaNETory (SE). The idea of SE is to force the model to focus on important information despite being presented with copious background distractions. We first identify the best interpretation loss function suited for bounding box-type object localization among multiple proposed variations. Then we train a CNN to incorporate this loss in its parameter optimization process using five different input resolution interpretation methods based on both class scores and probabilities. We show that SE improves the classification accuracy of the baseline model, with probability-based MaxPIn as the most successful interpretation method. Next, we design SE MINI as a smaller and faster version of SE based only on the correct class’ interpretation. We find SE MINI to be more stable than SE for probability interpretations at the expense of a bit smaller accuracy gain in general. SE MINI proves to be most effective with Expected Gradients probability interpretation. We also compare SE with other interpretation-aided training procedures and find that it mostly achieves better accuracy or gives a comparable result. Finally, we argue that the accuracy gain achieved by SE can serve as an objective evaluation metric for interpretation quality.