VIGIL-AI: Real-Time Cyber Threat Entity Recognition from OSINT Using Hybrid Regex–Transformer Models

Cybersecurity analysts increasingly rely on unstructured threat intelligence from social media, blogs, and open-source feeds, yet existing tools struggle to extract reliable indicators from such noisy sources. Rule-based methods achieve high precision on structured identifiers but miss contextual and obfuscated entities, while transformer-based language models provide contextual understanding but face adaptation challenges in specialized cybersecurity text. This paper introduces VIGIL-AI, a hybrid entity recognition pipeline for real-time cyber threat intelligence extraction from open-source intelligence (OSINT) sources. Real-time solutions such as VIGIL-AI are increasingly important in cybersecurity, where timely identification of threats can significantly reduce risk. A manually annotated dataset of 500 samples from Reddit, Twitter, and NewsAPI, combined with the publicly available bnsapa/cybersecurity-ner corpus, provides realistic training and evaluation data under noisy conditions. The system focuses on five key entity types relevant to threat intelligence: malware, vulnerabilities, indicators, systems, and organizations. Experimental results show that the hybrid approach improves coverage across both structured and informal mentions. DeBERTa-v3-large achieved the best balance of precision (0.678), recall (0.650), and F1 score (0.663), while the rule-based component ensured high accuracy on structured indicators. A proof-of-concept deployment, VIGIL-AI, demonstrates real-time extraction from live feeds, validating the system’s operational relevance. These findings indicate that hybrid methods, supported by realistic datasets and practical deployments, can strengthen automated threat intelligence workflows and help bridge the gap between academic evaluation and real-world cybersecurity applications.