Anomaly detection systems are crucial in cybersecurity for identifying unusual patterns or behaviors that may signify network attacks, particularly those that have not been seen before. The Converged Moth Search Algorithm (CMSA) and Attention-based Long Short-Term Memory (ALSTM) are proposed for IoT-based anomaly detection. The CMSA is used for feature selection which includes both quasi-oppositional mechanism and chaos theory. The quasi-oppositional mechanism is used to speed up the optimization process and enhance the feature exploration. The chaos theory is used to enhance the variety of populations and avoid local optimum issues. The ALSTM is used for classification which is utilized to overcome the vanishing and exploding gradient issues by concentrating on long and short-term dependencies. The three datasets such as BoT-IoT, IoT-23, and MQTT are used to detect anomalies. The min–max scaler is used for preprocessing which is helpful when feature data are on various scales. The recall, precision, f1-score, and accuracy are considered for evaluating CMSA-ALSTM performance. The CMSA-ALSTM attains the accuracy of 99.99, 99.99, and 95.76% for BoT-IoT, IoT-23, and MQTT datasets, respectively, when compared to dual Convolutional Neural Network (CNN) and Bidirectional LSTM (BiLSTM).