AI plays a key role in the future metaverse that pinpoints user experiences. Thus, the trustworthiness of such AI systems is vital, as failures in these systems can seriously harm the users. Existing works on trustworthy AI (TAI) metrics in the current cyberspace cannot cope with the user-centered metaverse, which is much more complicated in terms of system performance and user experience assessment. Thus, based on an overlook of fairness, privacy, and robustness, across the historical path from existing approaches, we propose a research agenda toward systematic & user-centered TAI metric taxonomy for the metaverse.