Unstructured data, such as textual data, is prevalent on digital platforms such as social media, with users posting content that sometimes includes sensitive information. For instance, in March of 2025, a press story by journalist Jeffrey Goldberg indicated that secret Trump administration plans for an air strike on Yemen had inadvertently been leaked to him ( U.S. national-security leaders included me in a group chat about upcoming military strikes in Yemen... ). Despite regulations, and the existence of access control policies data leaks such as this are persistent. One of the reasons for this is that contextually analysing unstructured data to determine whether or not sensitive information exists therein, is a challenging problem. Existing approaches identify individual elements of sensitive information such as Personally Identifiable Information (PII), but few works look at the issue of document sensitivity based on the occurrence of sensitive information. In this paper we propose a novel approach to classifying documents as either public or private, based on the occurrence of, and contextual interpretation of sensitive information within the document. We employ an ensemble model composed of transformer-encoder models and standard machine learning models to support the classification process. Our empirical study, conducted on a curated dataset composed of ENRON emails and Tweets of U.S. Congress members, indicates that the Random Forest algorithm when combined with the BERT model classifies all public documents correctly (i.e. 100% accuracy), and achieves a 0.98 recall score for private documents.