This paper presents an interesting parameter selection method for random forest algorithms, using a proprietary algorithm based on the concept of swarm optimization, called the Orange Cat Algorithm. The research was conducted on a smart home intrusion detection dataset, known as the Smart Home Intrusion Detection Dataset. In today’s world of increasing complexity of Internet of Things (IoT) systems, especially in the area of smart home security, there is a need for effective intrusion detection mechanisms. The Orange Cat algorithm is specifically designed to find the extremes of a function in a small number of iterations which is particularly useful for finding parameters for random forests, which contributes to improving the accuracy of intruder detection. Experiments showed that the proposed approach achieved an impressive accuracy of 99.40%. These results confirm the effectiveness of the Orange Cat Optimization algorithm in the parameter selection process, contributing to improved reliability and robustness of detection systems and thus preventing intrusions in IoT-based smart homes.