Rapid expansion of the Internet of Things (IoT) has introduced significant security challenges, particularly in mitigating botnet attacks. Traditional clustering techniques struggle to handle dynamic and high-dimensional IoT network traffic. This study proposes Dynamic Density-Based Ordering Points to Identify the Clustering Structure (DD-OPTICS), an enhanced density-based clustering algorithm for IoT botnet detection. DD-OPTICS dynamically adjusts clustering parameters based on network traffic density, improving the identification of anomalous patterns linked to botnet activities. The proposed approach leverages adaptive thresholding and distance-based clustering to ensure precise attack detection in real-time environments. Experimental results demonstrate the model’s effectiveness, achieving 96% accuracy, 94% precision, 95% recall, and 94% F1-score, as illustrated in the figure. These results highlight the robustness of DD-OPTICS in distinguishing between normal and malicious IoT traffic. The proposed approach surpasses traditional clustering methods, offering an effective and reliable solution for enhancing IoT network security against emerging cyberthreats.