Evaluating a Teaching Analytics Dashboard in Adult Education: Lessons Learned

Continuously monitoring student progress is predictive of student academic success but challenging for teachers in online courses. Although several teaching analytics dashboards (TADs) were developed, a lack of evidence of their impact and effectiveness on teaching and learning remains. Among the reasons stressed in earlier research, scholars pointed out: limited engagement of teachers in the design and development process of TADs, their busy schedules, their lack of data literacy, and neglect of pedagogical aspects. This paper presents a TAD which was developed to address the existing challenges, and its evaluation in higher education courses. Task-Technology Fit theory was used to evaluate teachers’ use of the TAD. The results show that the real-time TAD enables teachers to make in-time interventions to motivate low-performance students and to make changes in learning design while the course was occurring. Allowing teachers to use the TAD in running courses provides evidence of the tool use in a real-world context, which has been recognized as lacking in the field. Practical lessons are indicated for both the development process and the dashboard itself.