Deep learning is often called as a black-box model that does not give sufficient explainability for its prediction. Especially, in image regression problems such as image quality evaluation, it is not easy for deep learning models to give evidences for their prediction, resulting in the lack of explainability in AI. For this issue, this paper presents Positive-Negative Regression Activation Mappings (PN-RAM) that can visualize the positive and negative contributions in predictions by deep learning models for image regression problems. The proposed PN-RAM can visualize the positive and negative contributions in an image to a model prediction in a form of a saliency map obtained. In the experiments, we verify that PN-RAM can successfully visualize both positive and negative contributions of salient features in an image for image quality evaluation problems.