Since distal enhancers are involved in regulating target genes through physical contacting with proximal promoters, identifying enhancer-promoter interactions (EPIs) is critical to deepening our understanding of gene expression. However, high-throughput experimental methods for identifying EPIs are time-consuming and expensive. Therefore, computational methods for predicting EPIs would be valuable and important, but also face a lot of challenges. In this paper, we propose a novel deep learning-based method, namely EPIPAM, to predict EPIs only using genomic sequences. EPIPAM firstly uses a deep convolutional neural network to extract high-level sequence features, and then uses a position attention mechanism (PAM) to compute the positional correlation coefficients of two sub-regions separately coming from enhancers and promotors, aiming to focus on important regions of them. Benchmarking comparisons on six different cell lines show that EPIPAM performs better than the state-of-the-art methods in the task of EPIs prediction.