The Muse IT EU project is dedicated to delivering cultural heritage to people with sensory deprivation, relying on a complex approach to assistive technologies and virtual reality. Here, from the perspective of multisensory communication and interaction, we focus on one of its commitments, the generation of multimodal digital objects from well-known artefacts for virtual museums. Our current objective was to experiment with new methods and tools to remedy the partial loss of sight. Reporting work in progress, we created a scalable processing workflow to convert paintings into contoured multimodal digital objects with a simplified colour scheme, at the same time subjecting them to sonification and haptification. This was accomplished through a clearinghouse for both their non-semantic and semantic components by setting 60 RGB colours with the same number of piano notes and vibrotactile sensations on par. Respective tooling was augmented by topic-dependent soundscapes and their tactile parallels for subject contextualisation. Finally, the results were mapped to a knowledge graph covering content evolution of artefacts over time, thereby providing for the long-term digital preservation of artistic trends as well. Based on the WikiArt public dataset of paintings, we created and stored in Dataverse a test collection of 300 low-dimensional vs. 5 high-dimensional multimodal digital objects. These will be presented for user evaluation and technology co-design by user feedback in the final phase of the project.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Turning Paintings into Multimodal Digital Objects

  • Thomas Van Erven,
  • Sandor Darányi

摘要

The Muse IT EU project is dedicated to delivering cultural heritage to people with sensory deprivation, relying on a complex approach to assistive technologies and virtual reality. Here, from the perspective of multisensory communication and interaction, we focus on one of its commitments, the generation of multimodal digital objects from well-known artefacts for virtual museums. Our current objective was to experiment with new methods and tools to remedy the partial loss of sight. Reporting work in progress, we created a scalable processing workflow to convert paintings into contoured multimodal digital objects with a simplified colour scheme, at the same time subjecting them to sonification and haptification. This was accomplished through a clearinghouse for both their non-semantic and semantic components by setting 60 RGB colours with the same number of piano notes and vibrotactile sensations on par. Respective tooling was augmented by topic-dependent soundscapes and their tactile parallels for subject contextualisation. Finally, the results were mapped to a knowledge graph covering content evolution of artefacts over time, thereby providing for the long-term digital preservation of artistic trends as well. Based on the WikiArt public dataset of paintings, we created and stored in Dataverse a test collection of 300 low-dimensional vs. 5 high-dimensional multimodal digital objects. These will be presented for user evaluation and technology co-design by user feedback in the final phase of the project.