Generative AI offers potential in healthcare, especially through trustworthy and transparent virtual assistant solutions. Our D-STRESS research project, via its mental health use case, explores how to apply generative AI responsibly in high-stakes environments. By integrating explainable AI (XAI), multimodal longitudinal data, and human-centred design (HCD), we aim to support stress recovery by combining subjective self-reports, intervention reports, physiological signals, and contextual data to deliver a holistic understanding of stress patterns. Central to this effort is a clinician-facing dashboard co-designed with healthcare professionals. The dashboard presents interpretable and actionable insights using XAI techniques following a human-centred XAI (HCXAI) approach. It features a three-pane layout: patient data visualisation, analysis of model outputs, and a conversational agent for natural language interaction. This iterative, human-in-the-loop approach ensures task relevance and alignment with clinical workflows, promotes transparency, and fosters trust in AI-assisted decision-making.