Evaluating responses generated by large language models (LLMs) is challenging in the absence of ground-truth knowledge, particularly in specialised domains such as law. Increasingly, LLMs themselves are used to evaluate the responses they generate; however, this approach is prone to bias and inherent errors. To address these issues, we propose an unsupervised ensemble method that employs multiple general-purpose LLMs as a ‘collective judge’, rather than relying on a single model. Here we introduce a novel application of case alignment as an aggregation mechanism, achieving higher correlation with supervised metrics than unsupervised LLM-as-a-judge baselines. Specifically, we construct two spaces for the ensemble: one for reconstructed questions by the ensemble given the model’s original responses (‘problem-space’), and another for the set of answers generated in response to those reconstructed questions (‘solution-space’). By applying similarity-based alignment metrics across these two spaces, we gauge how closely our ensemble-based evaluation metric correlates with accuracy-based metrics that rely on ground-truth data. Our results on two legal Q&A datasets show significant correlations using this alignment strategy, suggesting that it can effectively evaluate LLM-generated responses even when ground truth is unavailable.