In the era of digital technology, it is crucial to prioritize privacy protection in healthcare. While Electronic Health Records text notes (EHR text notes) are widely used, securely removing patients’ private information from these records presents a significant challenge. This study investigates the utilization of a combination of large language models (LLM) and regular expression techniques to improve the extraction and standardization of Sensitive Health Information (SHI) from medical records sourced from Australian institutions. In response to the limitations identified in the initial Pythia language model, a hybrid approach was developed, incorporating both deep learning and regular expressions. The primary aim of the current study was to enhance the accuracy of SHI recognition in medical records. The study utilized 2294 records for training and 950 records for testing from the institutions, i.e., the University of New South Wales, the Lowy Cancer Research Centre, and the Health Science Alliance Biobank. The study employed PyTorch for model training and regular expressions for SHI identification. The results indicated that the hybrid approach highly improved the accuracy of SHI recognition, leading to more satisfactory and complete outcomes. This study contributes to the discussion on optimizing SHI recognition in healthcare datasets and emphasizes the synergy between deep learning and regular expressions. The findings have practical implications for refining healthcare information systems, highlighting the potential of combining advanced language models with rule-based approaches to enhance precision in medical data extraction and standardization.