The COVID-19 pandemic has demanded and encouraged the need for online education assessment that requires innovative solutions, and augmented reality (AR) could enhance the evaluation process. Using AR for developing innovative evaluation methods for online education assessment, comparing the effectiveness of AR-based approaches to traditional approaches, and examining the impact of AR on student engagement and motivation are the goals of this research paper. For this research project, we have selected 40 students of Diploma level from Universities in Nizwa. An AR-based intervention involving formative assessment using the high configuration of a Desktop computer & VR headset was implemented for group 1 (25) while group 2 (25) used a traditional teaching method with formative assessment. An analysis of covariance and a multivariate analysis of variance were conducted to analyze the data. Students’ learning performance and learning motivation were improved significantly by AR-VR-based formative assessment compared with traditional formative assessment. Further in this research project, we have included both quantitative and qualitative methods, such as surveys and analysis for the data collection. In this research project, we have used various tools and technologies such as RapidMiner, Python, and machine learning to find patterns and develop predictive models. Based on the findings of this study, educators and other stakeholders can gain insights into the potential benefits and challenges of using AR in online education assessment. Moreover, the study reviews the existing literature on AR and online assessment, presents case studies of successful implementation with the challenges in AR-based assessments, and highlights the advantages and best practices of AR-based assessments. Online education assessment methods based on AR hold a promising future for enhancing student engagement, improving evaluation accuracy, and adopting AR in assessments.