Graph Rewriting for User State-Based Dialogue Adaptation in Real-Time: An Application in Personalized Interview Training
摘要
Personalized learning is most effective as individuals have different learning styles and support needs. Many e-learning systems rely on static dialogue management, which offer limited flexibility to adapt to user behaviors and open discourse. This paper proposes a graph rewriting framework to adapt dialogue flow to user states in real-time. We propose its application in virtual job interviews—a key step toward employment—where personalized simulation training can be particularly valuable for individuals with diverse needs, including autistic users who may face challenges in social communication. Our framework leverages real-time multimodal data—speech, eye gaze, and physiology (e.g., heart rate)—to infer user states. We identified three key states in interviews: engagement, stress, and stuckness. Dialogue scripts are modeled as attributed, directed graphs, incorporating constraints to ensure consistency. Each interview question is represented as a node, with edges reflecting the flow of dialogue. The interview structure (host graph) was designed in collaboration with domain experts. Using graph rewriting, the framework dynamically modifies dialogue flows to enable contextually appropriate adaptations such as redirecting focus when the user goes off-topic, providing hints when stuck, or rephrasing questions for clarity—actions typically performed by coaches during mock interviews. Functionality of the grammar is demonstrated using interview scripts developed with experts, showcasing feasibility for virtual job interviews with simulated user states. This approach offers a personalized simulation experience that enhances realism, supports learning, and promotes the need for inclusive interviews.