Scholarlink

<p>Human Activity Recognition (HAR) has long been a fundamental research direction in the field of computer vision. Previous studies have primarily relied on traditional RGB cameras to achieve high-performance activity recognition. However, the challenging factors in real-world scenarios, such as insufficient lighting and rapid movements, inevitably degrade the performance of RGB cameras. To address these challenges, biologically inspired event cameras, with their advantages such as high dynamic range and high temporal resolution, offer a promising solution to overcome the limitations of traditional RGB cameras. In this work, we rethink human activity recognition by combining RGB and event cameras. We first publish the large-scale multi-modal RGB-Event aligned human activity recognition benchmark dataset, termed HARDVS 2.0, which bridges the dataset gaps, in terms of modality diversity and real-world scenario coverage. The existing mainstream HAR methods have been retrained and evaluated on this dataset to provide a new platform for comparison. It contains 300 categories of everyday real-world actions with a total of 107,646 paired videos covering various challenging scenarios. Inspired by the physics-informed heat conduction model, we propose a novel multi-modal heat conduction operation framework for effective activity recognition, termed MMHCO-HAR. More in detail, given the RGB frames and event streams, we first extract the feature embeddings using a stem network (embedding layer). Then, these feature embeddings are then processed by a set of multi-modal heat conduction blocks, where the core component is the Heat Conduction Operation (HCO) layer. In the HCO layer, we fuse RGB and event features through a multi-modal DCT-IDCT layer while adaptively incorporating the thermal conductivity coefficient via Frequency Value Embeddings (FVEs) into this module. After that, we propose an adaptive fusion module based on a policy routing strategy for high-performance classification. We conduct comprehensive experiments comparing our proposed method with baseline methods on the HARDVS 2.0 dataset and other public datasets, achieving Top-1 accuracies of 53.2% on HARDVS 2.0 and 57.4% on the PokerEvent dataset, outperforming existing models such as Vision mamba by +1.4% and +0.9%, respectively. These results demonstrate that our method consistently performs well, validating its effectiveness and robustness. The source code and benchmark dataset will be released on&#xa0;<a href="https://github.com/Event-AHU/HARDVS/tree/HARDVSv2">https://github.com/Event-AHU/HARDVS/tree/HARDVSv2</a>.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Human Activity Recognition using RGB-DVS Cameras: A Multi-modal Heat Conduction Model and A Benchmark Dataset

  • 期刊论文

  • 发表日期：

    2026年8月1日

查看全文
  • Shiao Wang,
  • Xiao Wang,
  • Bo Jiang,
  • Lin Zhu,
  • Guoqi Li,
  • Yaowei Wang,
  • Yonghong Tian,
  • Jin Tang

摘要

Human Activity Recognition (HAR) has long been a fundamental research direction in the field of computer vision. Previous studies have primarily relied on traditional RGB cameras to achieve high-performance activity recognition. However, the challenging factors in real-world scenarios, such as insufficient lighting and rapid movements, inevitably degrade the performance of RGB cameras. To address these challenges, biologically inspired event cameras, with their advantages such as high dynamic range and high temporal resolution, offer a promising solution to overcome the limitations of traditional RGB cameras. In this work, we rethink human activity recognition by combining RGB and event cameras. We first publish the large-scale multi-modal RGB-Event aligned human activity recognition benchmark dataset, termed HARDVS 2.0, which bridges the dataset gaps, in terms of modality diversity and real-world scenario coverage. The existing mainstream HAR methods have been retrained and evaluated on this dataset to provide a new platform for comparison. It contains 300 categories of everyday real-world actions with a total of 107,646 paired videos covering various challenging scenarios. Inspired by the physics-informed heat conduction model, we propose a novel multi-modal heat conduction operation framework for effective activity recognition, termed MMHCO-HAR. More in detail, given the RGB frames and event streams, we first extract the feature embeddings using a stem network (embedding layer). Then, these feature embeddings are then processed by a set of multi-modal heat conduction blocks, where the core component is the Heat Conduction Operation (HCO) layer. In the HCO layer, we fuse RGB and event features through a multi-modal DCT-IDCT layer while adaptively incorporating the thermal conductivity coefficient via Frequency Value Embeddings (FVEs) into this module. After that, we propose an adaptive fusion module based on a policy routing strategy for high-performance classification. We conduct comprehensive experiments comparing our proposed method with baseline methods on the HARDVS 2.0 dataset and other public datasets, achieving Top-1 accuracies of 53.2% on HARDVS 2.0 and 57.4% on the PokerEvent dataset, outperforming existing models such as Vision mamba by +1.4% and +0.9%, respectively. These results demonstrate that our method consistently performs well, validating its effectiveness and robustness. The source code and benchmark dataset will be released on https://github.com/Event-AHU/HARDVS/tree/HARDVSv2.