In the rapidly advancing field of Artificial Intelligence (AI) and AI-empowered software engineering, a fundamental tension exists between the autonomy of AI systems and the ethical imperatives that govern their development and application. This keynote addresses this tension by exploring how user-centered approaches can effectively reconcile AI autonomy with user needs and ethical integrity. As AI systems become more autonomous, ensuring that they adhere to ethical guidelines while meeting diverse user requirements presents a complex and significant challenge in software engineering. A critical research issue involves “translating” high-level ethical principles into concrete, actionable technical specifications that guide the entire software development lifecycle. This translation is complex, as ethical guidelines are often abstract and must be operationalized across all aspects of AI functionality, including data governance, privacy safeguards, user interface design, and decision-making algorithms. This challenge is particularly significant because traditional software engineering emphasizes reliability and accuracy, whereas AI components generate predictive hypotheses that may not always be accurate. Additionally, AI algorithms are trained on data, which makes their behavior dependent on the training data, further complicating the prediction of an AI-enabled system's precise functionality in all conditions. Through the application of theoretical frameworks and supported by practical case studies drawn from prior research, this keynote examines how these tensions impact the design and functionality of AI-empowered software. It explores the ethical implications of algorithmic decision-making, particularly where the complexity of AI processes can obstruct transparency and accountability. Additionally, it considers the user experience in interacting with autonomous systems, where ensuring trust and managing the balance between AI autonomy and human oversight are critical. These challenges are especially pronounced in domains where AI systems handle sensitive information, highlighting the need for strong ethical guidelines and technical solutions to ensure that AI autonomy aligns with societal values and operational transparency.