HYPERSEC: An Extensible Hypervisor-Assisted Framework for Kernel Rootkit Detection
摘要
Modern Endpoint Detection and Reaction (EDR) systems must remain reliable even when attackers acquire high privileges within the monitored operating system. Ideally, such systems should be protected from compromise, while maintaining deep visibility into the internal behavior of the target system. To meet this challenge, we aim to isolate the monitored system in a virtual machine (VM) and move the EDR logic into the hypervisor. We present HyperSec, a domain-specific language that allows the VM to safely delegate detection logic to the hypervisor by transmitting specialized monitoring programs. This design bridges the semantic gap by enabling the hypervisor to understand OS-level structures, while constraining accepted programs to protect the hypervisor against potential vulnerabilities. We evaluated our system against kernel rootkits that hide processes or elevate privileges and showed that HyperSec enables reliable detection of such threats. Our results indicate that the performance overhead remains acceptable, paving the way for broader adoption of hypervisor-based EDR with custom in-VM insight.