Domain coverage and criteria overlap across digital health technology quality assessments: a systematic review

Digital health technologies (DHTs) have the potential to improve health outcomes and increase health system efficiency. However, to obtain these benefits, it is necessary to distinguish between high- and low-quality DHTs across domains such as effectiveness, clinical safety, data privacy, and usability. For this purpose, numerous DHT quality assessments have been developed. This review aimed to examine the background characteristics, domain coverage, and criteria overlap of such assessments.

Assessment frameworks were identified through systematic searches of PubMed/MEDLINE, Embase and PsychINFO. Evaluation criteria of eligible assessments were extracted, and each criterion was assigned to one of twelve domains. The percentage of criteria included in each domain was compared across frameworks with different characteristics using Mann-Whitney U tests. Additionally, criteria overlap was examined among health system and government led third-party assessments.

The literature search identified 130 assessment frameworks that met eligibility criteria. Higher relative criteria coverage of the clinical safety and/or data privacy domains was observed for assessments that were published after rather than in or before 2016 (P_privacy= 0.030), and that were developed with rather than without patient (P_safety= 0.006; P_privacy= 0.034) or healthcare professional (P_safety= 0.005; P_privacy= 0.015) input. Among health system and government led third-party assessments, the highest median criteria overlap was observed in the credibility domain at 78.2%, followed by the data privacy domain at 64.0%. The domains of security and technical robustness demonstrated the lowest criteria overlap at 42.1% and 30.7%, respectively.

The observed differences in domain coverage based on stakeholder input and publication date highlight the importance of considering different perspectives and current best practices when developing and updating DHT quality assessments. Moreover, while the high criteria overlap found in some domains is encouraging, low overlap in other areas raises concerns regarding duplicated work and potential discrepancies in evaluation results, if a given DHT needs to meet varying standards across different assessments. To address these concerns, it would be beneficial for assessment owners to agree on a set of baseline criteria, which could be supplemented with separately-scored, context-specific criteria as needed. This would streamline evaluation processes and increase trust in assessment outcomes, thereby driving the adoption of high-quality DHTs and ultimately improving patient care quality and safety.