Rethinking the Evaluation of Scene Graph Generation
摘要
Scene Graph Generation (SGG) represents visual scenes by identifying entities and their relationships as triplets of subjects, predicates, and objects. Existing SGG evaluations primarily rely on Recall-based metrics, which exhibit deficiencies in comprehensively assessing model performance: 1) Such Recall-based metrics are restricted by incomplete triplet annotations, and 2) they treat all relationship types equally, ignoring the distinct semantic information conveyed by different relationships. To address these challenges, we propose two supplementary evaluation metrics, i.e., Semantic Triplet Precision (STP) and Semantic Information Depth (SID). The proposed metrics evaluate triplets beyond those explicitly annotated, taking both unannotated triplets and the varying informativeness of different predicates into account. To compute these supplementary metrics, we design a new SGG evaluation framework. In experiments, we reevaluate various SGG methods on the Visual Genome dataset and provide an extensive analysis and visualization of the proposed metrics. The experimental results fully reflect the completeness of the newly proposed metrics in the evaluation of scene graph generation methods.