In recent years, smart home technology has seen a significant surge in popularity, with smart speakers at the forefront of this technological revolution. These devices enhance everyday convenience by enabling users to control various home aspects, manage personal tasks, and access information through voice-activated interactions. Common functionalities include adjusting lighting, scheduling flights, and selecting music via voice commands. Despite their benefits, the widespread adoption of smart speakers has raised substantial privacy and security concerns. A primary issue is the lack of robust user authentication mechanisms, leaving these devices vulnerable to unauthorized access and control. To address these challenges, this project proposes a novel voice authentication system specifically designed for securing smart speakers. The system employs distinct wake-up words to activate and authenticate user interactions, thereby ensuring that only authorized users can access the functionalities of these devices. We conducted extensive testing of our prototype on standard hardware platforms such as laptops and Raspberry Pi, achieving an impressive 98% accuracy rate in recognizing and authenticating legitimate users. This paper discusses the development process, the technology underpinning our voice authentication model, and its potential for integration into existing smart speaker systems. Our results confirm the system’s viability for real-world application, effectively mitigating critical privacy and security risks associated with current smart speaker technologies.