This chapter explores the challenges associated with using maritime simulators for assessing seafarer competence, with a particular focus on the validity, reliability, and fairness of such evaluations. Key issues addressed include the potential for assessor bias and the inherent complexities of evaluating professional knowledge as it is enacted in dynamic, simulated environments. The chapter emphasizes the need for standardized assessment criteria, focused assessor training, and strategies to mitigate subjective evaluation bias. Drawing on insights from multiple studies, the chapter offers practical guidelines for enhancing the effectiveness of simulation-based competency testing. By bridging the gap between theoretical assessment principles and the practical realities of maritime training, the chapter provides a roadmap for developing fair and reliable assessment practices that reflect the challenges of evaluating professional knowledge in action.