Malware Classification Using Open Set Recognition and HTTP Protocol Requests

Malware is a serious threat to the modern Internet, as it is used to, e.g., sending spam or stealing bank login credentials. Typically, to communicate with the attacker, it utilizes popular network protocols such as the HyperText Transfer Protocol (HTTP). The network traffic characteristics related to this protocol can be used to detect malware and identify its family. The latter is a standard multi-class classification problem for which machine learning algorithms are utilized. However, existing methods cannot identify a real-world situation of encountering a new malware family, which was not known during their training phase. To address this issue, an Open Set Recognition (OSR) approach can be used, capable of a multi-class classification and identification of unknown class occurrence. In this paper, we apply OSR to the malware classification using HTTP requests and compare it with the existing solutions. In more detail, we analyze the classification performance of three OSR and two standard algorithms and their computation time. Additionally, we utilize two request representations: one based on Hfinger tool and the other relying on trigrams. The obtained experimental results allowed to select an optimal set of algorithms and HTTP request representations suitable for OSR scenarios.