Alternative polyadenylation (APA) is a critical process that enables genes to generate mRNA transcripts with different \(3'\) untranslated regions. Notably, during a transcription event, only one polyadenylation (poly(A)) site is used. Thus, estimating the relative usage of alternative poly(A) sites within a gene, known as the poly(A) site quantification problem, is crucial for unraveling the regulatory mechanisms of APA. However, existing approaches either frame the problem as a non-quantitative binary classification task or ignore the RNA structural information. To address these limitations, we propose a novel Hierarchical Attentive Graph Neural Network model for alternative poly(A) site quantification prediction, namely HAGAPS. To the best of our knowledge, we are the first to leverage Graph Neural Networks and RNA secondary structures to quantitatively predict the usage of multiple alternative poly(A) sites. In particular, our model employs a poly(A) site-level message passing network, incorporating RNA secondary structure information. In addition, to account for the competing interactions among poly(A) sites, HAGAPS integrates a gene-level message passing network combined with a nucleotide attention mechanism. Our experimental evaluation on publicly available datasets demonstrates that the proposed HAGAPS model significantly outperforms several state-of-the-art methods. Finally, for reproduction purposes, we make the implementation of HAGAPS publicly available at https://github.com/egiovanoudi/HAGAPS .