Internet of Things (IoT) is a rapidly growing technology that significantly benefits and impacts our daily lives. However, with the rise of IoT, new challenges in security have emerged. A formidable challenge to tackle new threats arises as a result of the constantly evolving nature of malware. In this paper, we present an anomaly detection system that has been integrated with a honeypot infrastructure to facilitate real-time data capture and anomaly detection. The two-layer anomaly detection system, named LAPIS, is capable of detecting malicious network traffic and identifying novel attacks. This integration aims to enhance security measures by providing a sophisticated mechanism for monitoring and analyzing network flows with precision and efficiency. We evaluated LAPIS using realistic network traffic collected by the honeypot during 12 months of operation. The experimental results show that the overall F1 score of LAPIS reaches 0.91 and 0.84 for detecting malicious network flows and zero-day attacks, respectively outperforming the closest state-of-the-art work. Compared to VirusTotal, which analyzes suspicious files and URLs to detect malware and malicious content, 61% of novel attacks are detected earlier by our system or yet to be available in VirusTotal.