<p>There is growing interest in designing Large Language Models (LLMs) to provide mental support for people navigating difficult life situations. However, evaluation remains a challenge for these applications. Manual evaluation is labor-intensive and time-consuming. Using LLMs themselves as automated evaluators, commonly referred to as LLM-as-a-Judge or LLMs-as-Judges, has been increasingly adopted, streamlining evaluative tasks while raising new questions about label quality, bias, and human oversight in the future of work. Despite their growing use, the abilities and limits of LLMs-as-Judges remain understudied in the mental support context. In this work, we fill this gap using real-world user-generated content from a Chinese mental support platform to evaluate the LLMs’ abilities and limits as evaluators. We found that LLMs-as-Judges achieved 72–84% agreement with human judges when assessing the quality of pairs of human-generated responses. The agreement dropped to 57–70% when assessing LLM–LLM pairs, and further reduced to 26–60% for human–LLM pairs. While task difficulty contributed to this decline, strong self-enhancement bias, i.e., the tendency of LLMs-as-Judges to favor LLM-generated responses, also played a big role. Simply switching LLM models between generating and judging did not sufficiently reduce this bias. Thus, our work cautions against the over-reliance on LLMs-as-Judges in the development and evaluation of mental-support LLM applications, highlighting the criticality of human oversight in the workforce. We further call on future research to address the self-enhancement bias inherent in LLMs-as-Judges to promote their objective and responsible use in the future of work.</p>