Ensuring compliance with data privacy regulations and policies poses a challenge for organizations. Natural Legal Language Processing (NLLP) using deep learning presents a promising avenue for addressing this challenge. A critical hindrance to fully harnessing the potential of machine learning techniques in privacy compliance is the limited availability of extensive, domain-specific labeled datasets. In response to this critical gap, this paper introduces GDPR120Q, a new specialized, expert-annotated question-answering dataset, which we used to test for formal compliance of privacy documents with the European Union General Data Protection Regulation (GDPR). GDPR120Q consists of 39,834 annotations, 120 recent privacy policies, 108 possible labels and is, to date, the only open-source large-scale dataset dealing with privacy issues under European law. We leverage this dataset to identify text spans in privacy policies that require attention from individuals and data processors for compliance purposes. We test the dataset with baseline transformer models as well as GPT3, which return promising results but leave substantial room for improvement. As one of the only large, specialized NLP annotated by experts, GDPR120Q can contribute to the development of innovative approaches to privacy compliance and serves as a benchmark for the NLLP community.