Current approaches to evaluating recommendation explanations in industrial platforms focus solely on persuasiveness–a metric that often serves the platform’s own interests, such as driving clicks–while largely ignoring whether these explanations genuinely benefit users’ decision-making. This narrow focus limits long-term value of explanations for end-users. In this paper, we advocate shifting toward a user-centric paradigm and propose a framework assessing three key properties: Usefulness, Fidelity, and Generalizability. These capture essential aspects of explanation quality and are amenable to large-scale quantitative measurement. Using a real-world Meituan dataset, we demonstrate the discriminative and diagnostic power of our metrics and the framework’s comprehensiveness. A user study further examines how these objective metrics align with subjective user perceptions.