A Comparative Analysis of Model Alignment Regarding AI Ethics Principles

As LLMs gain an increasingly relevant role and agency, their alignment with human values, principles and goals is crucial for their responsible deployment and acceptance. The main goal of this study is to assess the alignment of different LLMs regarding the relative importance of AI Ethics principles across different domains. To this end, human experts in different domains were asked, through a questionnaire, to rate the relative importance of six AI Ethics principles in their respective domains, totaling 6 domains. Then, five publicly available LLMs were asked to rate the same Ethics principles in different domains. Multiple prompts were used multiple times, to also evaluate consistency, totaling 90 runs per LLM. Model alignment was measured through the correlation with human experts, and consistency was evaluated through the standard deviation. Results show varying degrees of alignment and consistency, with a couple of models showing satisfactory results. This makes it possible to envisage the use of such models to automatically configure and adapt data pipeline ecosystems and architectures across different domains, selecting processes, dashboard elements or monitored KPIs according to the target domain or the goals of the system.