In today’s society the increasing dependence on communication technology highlights the importance of having methods for examining network traffic to tackle the continuously changing threats. In this paper, we explore the approaches and tools used in investigating network traffic with the goal of empowering individuals to protect their networks from various attacks and malicious software. Furthermore, we discuss how attackers use masking techniques to avoid detection highlighting the need for enhancing network security protocols. Although identifying malware is the main concern of our proposed approach. We suggest an algorithm that can also identify insider threats and unusual network behavior. With the assistance of advanced filtering, hash analysis, and indication of compromise techniques, we can quickly recognize and prevent possible threats. Our defense against such threats is further strengthened by sharing hashes that have been found with antivirus software. Our algorithm shows great utility in detecting Windows-based malware, as demonstrated by testing it on infected packet capture files.