Information systems (IS) are increasingly being used to support more complex and knowledge-intensive decision-making tasks. Contemporary systems often rely on machine learning, artificial intelligence (AI), and deep learning methods. While these technologies are promising in processing complex problems and large datasets, their black-box nature poses challenges for these systems’ adoption in organizational settings, where understanding the rationale used by AI-driven decision-support tools is important. This study examines supply chain experts’ interactions with AI systems through the lens of the situation awareness (SA) model. We identify factors leading to a user’s decision to either delegate decision-making tasks to AI or intervene in the AI-driven processes. Our findings suggest that mismatches between the AI’s SA and the expert user’s SA create tensions that impact the delegation of tasks to the AI system. We discuss additional sources of tension and propose human-centered design practices to mitigate these issues.