The rapid proliferation of Internet of Things (IoT) devices has introduced significant security challenges, particularly in ensuring the reliability and resilience of RFID systems, a critical component of IoT architectures. Among these challenges, denial of service (DoS) attacks pose a significant threat, potentially disrupting operations and compromising sensitive data. This study proposes a comprehensive framework that integrates lightweight cryptographic techniques and machine learning (ML)-based algorithms for enhanced RFID security. The research begins with an analysis of IoT security challenges, focusing on the vulnerabilities of RFID systems and the need for resource-efficient cryptographic solutions. Lightweight cryptography (LWC) is employed to ensure secure and efficient data encryption, while ML-based authentication and anomaly detection algorithms are utilized to identify unauthorized access and mitigate DoS attacks. The proposed system is modeled and evaluated using a layered RFID architecture and a detailed flowchart that incorporates data decryption, feature extraction, and real-time anomaly detection. By leveraging state-of-the-art ML algorithms such as Isolation Forest, the framework ensures robust authentication and anomaly detection capabilities. The results demonstrate the feasibility of combining lightweight cryptography with ML techniques to address IoT security challenges effectively, providing a scalable and resource-efficient solution for safeguarding RFID systems against evolving cyber threats.