Identifying meaningful and sensitive information from CCTV footage is challenging due to the tremendous amount of data being generated. In addition, storing this huge amount of video requires a significant amount of space. In this context, Knowledge Graph (KG) can represent salient information present in video and can be queried to retrieve events, actions, anomalies, and know-how. However, constructing a KG from video is challenging. Therefore, this study proposes a systematic approach that converts video data into textual descriptions by selecting keyframes based on spatial relationships, followed by the extraction of triplets to construct a knowledge graph. The captioning phase identifies frames that feature interaction with objects and produces descriptions in natural language. These descriptions are subsequently analyzed to derive triplets, combining custom refinements to manage compound entities and relational expressions, resulting in data suited for knowledge representation. This approach enhances data retrieval from extensive video archives, delivering practical value for surveillance operations and automated documentation.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

From Pixels to Triplets: Building Knowledge Graph from Surveillance Video

  • Atoholi V. Chisho,
  • Ashish Singh Patel

摘要

Identifying meaningful and sensitive information from CCTV footage is challenging due to the tremendous amount of data being generated. In addition, storing this huge amount of video requires a significant amount of space. In this context, Knowledge Graph (KG) can represent salient information present in video and can be queried to retrieve events, actions, anomalies, and know-how. However, constructing a KG from video is challenging. Therefore, this study proposes a systematic approach that converts video data into textual descriptions by selecting keyframes based on spatial relationships, followed by the extraction of triplets to construct a knowledge graph. The captioning phase identifies frames that feature interaction with objects and produces descriptions in natural language. These descriptions are subsequently analyzed to derive triplets, combining custom refinements to manage compound entities and relational expressions, resulting in data suited for knowledge representation. This approach enhances data retrieval from extensive video archives, delivering practical value for surveillance operations and automated documentation.