In this study, we investigate user interactions on an Artificial Intelligence (AI)-enabled platform, featuring a chatbot designed to augment user engagement in learning how to solve a puzzle. Our platform’s core AI algorithm leverages macro-actions-sequences of moves, not interpretable or usable to users, essentially functioning as a ‘black box.’ To address this, we employed scaffolding design strategies and explainable AI principles to develop an innovative conversational user interface (UI). Utilizing eye-tracking techniques, we collected users’ gaze data to assess their gaze patterns and attention distribution during a problem-solving process. By focusing on gaze metrics, e.g., fixation duration, saccade frequency, area of interest, and heatmap, we found users’ visual attention correlates with scaffolding-enabled UI elements, and thus positively influencing their problem-solving experience. This preliminary study allows us to evaluate the UI's intuitiveness and identify design strategies for user engagement improvement, adaptable to diverse learning styles, and thus potentially enhancing problem-solving efficiency on our AI platform.