Cultural heritage institutions (GLAM) have a societal role of guaranteeing access to their collections independently of location or background, as a pledge for knowledge equity. However, around one billion people (15% of the world’s population) experience some form of disability that hinders such access, especially when considering the multimedia gap: differences in the use of different types of media to convey content to different audiences. Knowledge Graphs are increasingly becoming more multimodal by supporting images and text. However, the extent to which they address the multimedia gap for people with disabilities is not well understood, due to the lack of appropriate cultural evaluation frameworks. To address this, here we propose CUBE-MT, a benchmark and dataset that leverages generative models to build Multimodal Knowledge Graphs (MMKGs) with surrogate, multimedia representations, that adapt to the sensory capacities of cultural heritage collection users. We extend the CUBE (CUltural BEnchmark for Text-to-Image models) and Muse-IT datasets(MuseIT is a Horizon Europe project on innovative technologies for broadening access to cultural heritage for people with disabilities: https://www.muse-it.eu/ ) for paintings to encompass 6 modalities (text, images, Braille, speech, music, and 3D models); a collection of prompts to account for their cultural awareness and diversity; and a dataset with the resulting MMKG mapped to Wikidata. We show usage and evaluate the effectiveness of our approach in: (1) a quantitative assessment of cultural diversity; (2) an expert survey; and (3) a user study with people with aphasia focusing on perceptual and comprehension differences between model-generated and original MMKG objects.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

CUBE-MT: A Cultural Benchmark for Multimodal Knowledge Graph Construction with Generative Models

  • Albert Meroño-Peñuela,
  • Xin Fan Guo,
  • Nitisha Jain,
  • Filip Birčanin,
  • Timothy Neate,
  • Thomas van Erven,
  • Sándor Daranyi,
  • Nasrine Olson

摘要

Cultural heritage institutions (GLAM) have a societal role of guaranteeing access to their collections independently of location or background, as a pledge for knowledge equity. However, around one billion people (15% of the world’s population) experience some form of disability that hinders such access, especially when considering the multimedia gap: differences in the use of different types of media to convey content to different audiences. Knowledge Graphs are increasingly becoming more multimodal by supporting images and text. However, the extent to which they address the multimedia gap for people with disabilities is not well understood, due to the lack of appropriate cultural evaluation frameworks. To address this, here we propose CUBE-MT, a benchmark and dataset that leverages generative models to build Multimodal Knowledge Graphs (MMKGs) with surrogate, multimedia representations, that adapt to the sensory capacities of cultural heritage collection users. We extend the CUBE (CUltural BEnchmark for Text-to-Image models) and Muse-IT datasets(MuseIT is a Horizon Europe project on innovative technologies for broadening access to cultural heritage for people with disabilities: https://www.muse-it.eu/ ) for paintings to encompass 6 modalities (text, images, Braille, speech, music, and 3D models); a collection of prompts to account for their cultural awareness and diversity; and a dataset with the resulting MMKG mapped to Wikidata. We show usage and evaluate the effectiveness of our approach in: (1) a quantitative assessment of cultural diversity; (2) an expert survey; and (3) a user study with people with aphasia focusing on perceptual and comprehension differences between model-generated and original MMKG objects.