Sysalign: protect system calls via semantic alignment of critical paths and syscall sequences
摘要
System call security is crucial for host-based intrusion detection, as security-sensitive system calls (e.g., execve, mprotect) play a vital role in completing attacks. However, existing defense methods face significant limitations. Static analysis and system call filtering cannot prevent the malicious use of necessary syscalls, while runtime methods based on control-flow, syscall sequences, or parameter integrity are vulnerable to sophisticated attacks, such as data-only attacks, that mimic benign execution patterns. In this paper, we propose a novel contrastive learning framework, SysAlign, for system call security that ensures the semantic consistency between a system call’s macro-level and micro-level intents. SysAlign learns the macro-level intent by the sequence of system calls, which reflects the broader objective of a program phase. The micro-level intent is derived from the specific execution path, capturing the detailed execution context of system calls. Malicious system calls disrupt this consistency, leading to a detectable misalignment between the two intents. To minimize the overhead, we efficiently represent micro-level intent using critical points instead of tracking full execution paths. These points are extracted via dominator tree analysis on the program’s over-approximated control-flow graph, effectively balancing rich semantics with performance. Additionally, to address the scarcity of attack data, our framework incorporates a tailored negative sampling strategy, enhancing the model’s robustness. We evaluated SysAlign on real-world applications, including Nginx, NullHttpd, cURL, and SQLite3. The results demonstrate that our method effectively detects a diverse range of malicious system call behaviors, including those resulting from control-flow hijacking and data-only exploits. Our approach achieves an F1-score exceeding 96% with a performance overhead below 7%. This work presents a practical and effective advancement in system call security, significantly outperforming existing techniques.