The growth in network traffic and the increasing complexity of cyber threats necessitate robust systems for detecting anomalies that indicate security breaches. This research presents a methodology for finding anomalies in packets sent when the connection is established. It uses a machine learning model and a packet sniffer. It captures Transmission Control Protocol (TCP), User Datagram Protocol (UDP), IPv4, and Internet Control Message Protocol (ICMP) segments to predict if any anomalies are present (Sanders in Practical packet analysis: using wireshark to solve real-world network problems, No Starch Press, San Francisco, 2017). An unsupervised learning model is utilized. The presence of unlabeled data to enhance the real-time prediction using isolation forest model. The data collected by packet sniffer undergoes avoiding null values and encoding addresses, and thus an isolation forest is used so that it predicts if anomalies are present using binary trees. The performance is evaluated on the basis of metrics like accuracy, precision, and F1-score (Goutte and Gaussier in European conference on information retrieval, Springer, New York, 2005). The result illustrates the model is accurate in predicting whether anomalies are present. Future work is focused on enhancing the model’s capabilities with more protocols and an active defense mechanism. The study addresses real-world deployment challenges especially in heterogeneous environments like IoT-based networks. While isolation forest is getting high accuracy, future research could explore hybrid approaches combining traditional statistical methods with deep learning techniques for enhanced industry applications (Ahmed et al. in J Netw Comput Appl 60:19–31, 2016).