Scene Graph Generation (SGG) represents visual scenes by identifying entities and their relationships as triplets of subjects, predicates, and objects. Existing SGG evaluations primarily rely on Recall-based metrics, which exhibit deficiencies in comprehensively assessing model performance: 1) Such Recall-based metrics are restricted by incomplete triplet annotations, and 2) they treat all relationship types equally, ignoring the distinct semantic information conveyed by different relationships. To address these challenges, we propose two supplementary evaluation metrics, i.e., Semantic Triplet Precision (STP) and Semantic Information Depth (SID). The proposed metrics evaluate triplets beyond those explicitly annotated, taking both unannotated triplets and the varying informativeness of different predicates into account. To compute these supplementary metrics, we design a new SGG evaluation framework. In experiments, we reevaluate various SGG methods on the Visual Genome dataset and provide an extensive analysis and visualization of the proposed metrics. The experimental results fully reflect the completeness of the newly proposed metrics in the evaluation of scene graph generation methods.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Rethinking the Evaluation of Scene Graph Generation

  • Jingyi Wang,
  • Hanwei Gao,
  • Zhidong Deng

摘要

Scene Graph Generation (SGG) represents visual scenes by identifying entities and their relationships as triplets of subjects, predicates, and objects. Existing SGG evaluations primarily rely on Recall-based metrics, which exhibit deficiencies in comprehensively assessing model performance: 1) Such Recall-based metrics are restricted by incomplete triplet annotations, and 2) they treat all relationship types equally, ignoring the distinct semantic information conveyed by different relationships. To address these challenges, we propose two supplementary evaluation metrics, i.e., Semantic Triplet Precision (STP) and Semantic Information Depth (SID). The proposed metrics evaluate triplets beyond those explicitly annotated, taking both unannotated triplets and the varying informativeness of different predicates into account. To compute these supplementary metrics, we design a new SGG evaluation framework. In experiments, we reevaluate various SGG methods on the Visual Genome dataset and provide an extensive analysis and visualization of the proposed metrics. The experimental results fully reflect the completeness of the newly proposed metrics in the evaluation of scene graph generation methods.