We present a tool that leverages a Multimodal Large Language Model (MLLM) for the automatic classification of human activities from images of urban scenes. Starting from an image of spaces populated with people, the tool is capable to classify them according to five features: age group, sex, bodily posture, activity level and social configuration. The tool implements a sequential pipeline consisting of Faster R-CNN for person detection, followed by postprocessing and two consecutive applications of GPT-4o models for refined image description and information extraction. In the paper we also present an experimental test used to preliminary validation of the tool, comparing the ground truth on 24 images of urban scenes with the estimates provided by the tool, yielding a good degree of alignment. The tool is part of the wider research programme of massive assessment of urban affordances, within the framework of the capability approach.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Classifying Human Activities in Urban Spaces with a Multimodal AI: Towards a Massive Assessment of Urban Affordances

  • Ivan Blečić,
  • Alessandro Floris,
  • Giulia Giliberto,
  • Giuseppe A. Trunfio

摘要

We present a tool that leverages a Multimodal Large Language Model (MLLM) for the automatic classification of human activities from images of urban scenes. Starting from an image of spaces populated with people, the tool is capable to classify them according to five features: age group, sex, bodily posture, activity level and social configuration. The tool implements a sequential pipeline consisting of Faster R-CNN for person detection, followed by postprocessing and two consecutive applications of GPT-4o models for refined image description and information extraction. In the paper we also present an experimental test used to preliminary validation of the tool, comparing the ground truth on 24 images of urban scenes with the estimates provided by the tool, yielding a good degree of alignment. The tool is part of the wider research programme of massive assessment of urban affordances, within the framework of the capability approach.