Least Information Redundancy Algorithm of Printable Shellcode Encoding for X86

Shellcode is a critical element in computer security that exploits vulnerabilities within software systems. Shellcode is written in machine code and often designed to be compact in size, evading detection by security software. Printable shellcode, specifically, comprises only printable ASCII characters (0x21–0x7E), including letters, numbers, and punctuation marks. The key advantage of printable shellcode lies in its ability to be embedded within data streams, which may undergo parsing or manipulation by applications that would otherwise filter or modify non-printable characters. The prevalent methods for generating printable shellcode involve encoding algorithms, such as the Riley Eller algorithm (integrated into Metasploit). However, previous research on printable shellcode has primarily focused on the availability and reduction of the encoded shellcode’s size, without adequately considering the constraint imposed by the information entropy of the encoding algorithm within the context of printable shellcode. In this paper, we demonstrate the existence of minimal information redundancy in printable shellcode. Subsequently, we introduce Lycan, an implementation of a novel algorithm that surpasses previous encoding algorithms in terms of the size efficiency of the encoded shellcode. Lycan achieves the least theoretical information redundancy. Through experimentation, we observe that Lycan generates the most compact shellcode among existing tools when the shellcode’s size exceeds a certain threshold.