The paper examines the effectiveness of Generative Artificial Intelligence (GenAI) models in evaluating translation quality in a low-resource language, focusing on Slovak. With the growing use of GenAI in commercial machine translation systems, there is a need to understand whether these models can support translation education. The study compares GenAI generated evaluations against student assessments across four error categories: accuracy, style, terminology, and language. Evaluations were conducted under two conditions: using only the source and machine translation output, and with an additional human reference translation. Results show model performance varies by model and category, with Claude demonstrating the highest alignment overall. The inclusion of a human reference did not consistently improve performance. These findings suggest that GenAI can support translation education but must be applied with caution, particularly in low-resource language settings.