Advanced artificial intelligence techniques, including metaheuristic algorithms, have demonstrated exceptional effectiveness in the field of transportation by delivering near-optimal solutions to complex real-world optimization problems much faster than traditional approaches. While metaheuristic algorithms can outperform traditional methods, the rapid development of new metaheuristics makes it challenging to identify the most suitable one for solving specific problems. Therefore, determining the appropriate evaluation measures is essential for selecting the best metaheuristic. In this study, we address this issue by conducting a review on performance measures for metaheuristics in solving the activity chain optimization problem (ACOp), which is too elaborate for traditional algorithms to solve efficiently within an acceptable computational time frame. The research findings categorize these measures into two groups: efficiency and effectiveness. Likewise, our results indicate that the most suitable efficiency measures for assessing ACOp are the rate of convergence, diversity, and computation costs. For effectiveness measures, successful convergence, scalability, and Bayesian tests are identified as most appropriate. Defining these metrics is beneficial for practitioners and modelers as it standardizes the algorithmic evaluation process from an early stage.