Benchmarking Trust: A Metric for Trustworthy Machine Learning

In the evolving landscape of machine learning research, the concept of trustworthiness receives critical consideration, both concerning data and models. However, the lack of a universally agreed upon definition of the very concept of trustworthiness presents a considerable challenge. The lack of such a definition impedes meaningful exchange and comparison of results when it comes to assessing trust. To make matters worse, coming up with a quantifiable metric is currently hardly possible. In consequence, the machine learning community cannot operationalize the term, beyond its current state as a hardly graspable concept. This contribution is the first to propose a metric assessing the trustworthiness of machine learning models and datasets. Our FRIES Trust Score is grounded in five key aspects we understand to be the fundamental building blocks of trust in machine learning – fairness, robustness, integrity, explainability, and safety. We evaluate our metric across three datasets and three models, probing the metric’s reliability by enlisting the expertise of ten machine learning researchers in its application. The results underline the usefulness and reliability of our method, seeing distinct overlaps between the participants’ ratings.