Benchmarking the User eXperience and Usability of Online Social Networks: Proposal of an Evaluation Framework

The ubiquity of Information and Communication Technologies (ICT) presents challenges in designing experiences and interfaces. Hence, it becomes crucial to have solutions that facilitate the comparison of various products, discerning their relative strengths and weaknesses. Therefore, this paper presents a proposal of an Evaluation Framework for Benchmarking the user eXperience and Usability of Online Social Networks. Its goal is to provide a holistic perspective of user eXperience and satisfaction, checked in conformance with a detailed heuristic analysis. This Framework is divided into three parts, comprising (i) the selection of experts, and products to benchmark; (ii) the evaluation of UX Satisfaction of each product with Emocards, and (iii) test the User eXperience within a Usability Heuristic Evaluation for each product. By applying the specific study case of seven senior online social networks and resorting to 14 experts in the field of ICT, it was possible to build a holistic Benchmarking Evaluation Framework. In particular, this Framework introduces new perspectives on (i) the priority sequence for UX and satisfaction, followed by heuristic detail; (ii) the seamless incorporation of feedback obtained during the think-aloud process into the analysis and data representation; (iii) the creation of a normalized score for each heuristic of every product, thus allowing for a direct comparison between products; and (iv) the creation of integrated representations of the two dimensions under study – i.e., UX satisfaction and Usability Heuristic Evaluation. Overall, we hope that this proposed Framework can serve as a simple yet effective approach to represent and compare evaluation results between products.