Code review automation is an important area of research given the time and effort required by traditional processes. Evaluating comment generation models requires automated metrics that quantify the similarity between generated and human-authored comments. Prior work has relied on lexical metrics like BLEU and ExactMatches, which fail to capture semantic differences when phrasing varies. Sentence embedding models offer a promising alternative; however, so far there has been no evidence of their superiority in capturing semantic similarity on code review data. In an empirical study with human subjects, we use OpenAI embeddings to assess semantic similarity in review comments. Our analyses show that the embedding-based metric aligns more closely with human annotations than BLEU. Furthermore, our method to support metric choices with empirical evidence may represent a significant advancement in the domain.