<p>Human Action Recognition (HAR) stands as a pivotal research domain in both computer vision and artificial intelligence, with RGB cameras dominating as the preferred tool for investigation and innovation in this field. However, in real-world applications, RGB cameras encounter numerous challenges, including light conditions, fast motion, and privacy concerns. Consequently, bio-inspired event cameras have garnered increasing attention due to their advantages of low energy consumption, high dynamic range, etc. Nevertheless, most existing event-based HAR datasets are low resolution (<InlineEquation ID="IEq1"> <EquationSource Format="TEX">\(346 \times 260\)</EquationSource> <EquationSource Format="MATHML"><math> <mrow> <mn>346</mn> <mo>×</mo> <mn>260</mn> </mrow> </math></EquationSource> </InlineEquation>). In this paper, we propose a large-scale, high-definition (<InlineEquation ID="IEq2"> <EquationSource Format="TEX">\(1280 \times 800\)</EquationSource> <EquationSource Format="MATHML"><math> <mrow> <mn>1280</mn> <mo>×</mo> <mn>800</mn> </mrow> </math></EquationSource> </InlineEquation>) human action recognition dataset based on the CeleX-V event camera, termed CeleX-HAR. It encompasses 150 commonly occurring action categories, comprising a total of 124,625 video sequences. Various factors such as multi-view, illumination, action speed, and occlusion are considered when recording these data. To build a more comprehensive benchmark dataset, we report over 20 mainstream HAR models for future works to compare. In addition, we also propose a novel Mamba vision backbone network for event stream based HAR, termed EVMamba, which equips the spatial plane multi-directional scanning and a novel voxel temporal scanning mechanism. By encoding and mining the spatio-temporal information of event streams, our EVMamba has achieved favorable results across multiple datasets. Both the dataset and source code have been released on <a href="https://github.com/Event-AHU/CeleX-HAR">https://github.com/Event-AHU/CeleX-HAR</a>.</p>

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Event Stream based Human Action Recognition: A High-Definition Benchmark Dataset and Algorithms

  • Xiao Wang,
  • Shiao Wang,
  • Pengpeng Shao,
  • Lin Zhu,
  • Bo Jiang,
  • Yonghong Tian

摘要

Human Action Recognition (HAR) stands as a pivotal research domain in both computer vision and artificial intelligence, with RGB cameras dominating as the preferred tool for investigation and innovation in this field. However, in real-world applications, RGB cameras encounter numerous challenges, including light conditions, fast motion, and privacy concerns. Consequently, bio-inspired event cameras have garnered increasing attention due to their advantages of low energy consumption, high dynamic range, etc. Nevertheless, most existing event-based HAR datasets are low resolution ( \(346 \times 260\) 346 × 260 ). In this paper, we propose a large-scale, high-definition ( \(1280 \times 800\) 1280 × 800 ) human action recognition dataset based on the CeleX-V event camera, termed CeleX-HAR. It encompasses 150 commonly occurring action categories, comprising a total of 124,625 video sequences. Various factors such as multi-view, illumination, action speed, and occlusion are considered when recording these data. To build a more comprehensive benchmark dataset, we report over 20 mainstream HAR models for future works to compare. In addition, we also propose a novel Mamba vision backbone network for event stream based HAR, termed EVMamba, which equips the spatial plane multi-directional scanning and a novel voxel temporal scanning mechanism. By encoding and mining the spatio-temporal information of event streams, our EVMamba has achieved favorable results across multiple datasets. Both the dataset and source code have been released on https://github.com/Event-AHU/CeleX-HAR.