Software vulnerabilities are referred to as weaknesses in the source code. Hackers exploit these weaknesses to perform malicious actions, including accessing sensitive data and injecting a computer virus to hijack the computer system. Identifying these vulnerabilities is challenging. Even a perfectly functional program may have hidden vulnerable patterns. Contrastingly, locating these patterns manually by cybersecurity experts is onerous and time-consuming. Existing research uses deep learning algorithms to automate the process of finding vulnerable patterns. However, extracting relevant features from the source code is challenging, as source code can exceed hundreds of lines. Existing researchers either extract individual functions in a program or encapsulate the entire program, leading to under- or over-representation, respectively. A recent notion of using program slices has emerged, which only retains the most likely vulnerabilities causing statements. However, no significant research has been conducted to identify the most significant characteristics that are likely to cause vulnerabilities. Therefore, in this research, we investigate statistical heuristics to dynamically determine the most representative vulnerability characteristics and propose a deep neural network based on RoBERTa embeddings, which is fine-tuned on CodeBERT using multi-sample dropout to enhance generalizability. Our experimental results on real-world software source databases show up to 37% (F1 score) improvements in vulnerability prediction, with 61% reduced training time requirements when compared with the state-of-the-art baselines (VulDeePecker, SySeVR, and SlicedLocator).