Large Language Models (LLMs) are emerging as promising tools for generating content, such as educational content for computer science education. When using LLMs, it is necessary to evaluate their outputs to ensure that the quality meets the required standards. This study explores the performance of two LLMs in evaluating the quality of existing Bebras tasks by comparing their evaluations to those made by human experts from the Bebras community. The evaluation is based on predefined criteria such as relevance to informatics, clarity, and learning experience. The study analyzes three tasks of varying difficulty levels aimed at 12–14-year-olds and examines differences in evaluations based on the roles of human experts (teachers, researchers, and organizers) and the corresponding roles prompted to the LLMs. The results indicate varying degrees of alignment between LLMs and human experts, with LLMs struggling to evaluate the learning experience and the time needed to solve the tasks. Additionally, an interesting phenomenon was observed among human expert participants: there was a lack of consensus in estimating the difficulty levels of the tasks. This highlights the inherent subjectivity in difficulty assessment and poses challenges for both human evaluators and AI models. This approach provides insights into the potential of LLMs to support the Bebras community in evaluating tasks and potentially generating new Bebras tasks in the future.