This paper examines three multimodal data fusion techniques. Multimodal Data Fusion–based Graph Contrastive Learning (MDFCL), Graph-Structured & Interlaced-Masked Fusion Network (GSIFN), and Perceiver IO—on agricultural time-series data. MDFCL builds individual graphs for each modality and applies unsupervised contrastive losses to align the embeddings of nodes, inducing cross-modal robustness (achieving 97.66% accuracy with image-only inputs). GSIFN develops an interlaced masking joint Transformer, capturing higher-order interactions with efficiency, along with self-supervised LSTM-based side tasks to counter redundancy (achieving 100.00% classification accuracy in rigorous 5-fold cross-validation). Perceiver IO adopts an implicit-latent bottleneck Transformer, providing heterogeneous streams of agricultural data flexibly with near-linear complexity without individual encoders per modality. Perceiver IO also achieved a similar result as compared to MDFCL (achieving 97.66% accuracy) while it has fewer parameters and requires less computations. Although these models have been found to be successful with generic multimodal tasks, their applicability to fusion of agro-sensor and growth-image data has thus far remained relatively less explored. We evaluate their adaptation versatility, advantages, and limitations in this study, providing actionable recommendations on fusion of agricultural image and time-series data to support precision agriculture with robustness, scalability, and efficiency.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Evaluating Multimodal Fusion Strategies for Resilient Agricultural Sensing Systems

  • Ponnuri Aniruddha,
  • Abhay Shaji Valiyaparambil,
  • K. Sornalakshmi

摘要

This paper examines three multimodal data fusion techniques. Multimodal Data Fusion–based Graph Contrastive Learning (MDFCL), Graph-Structured & Interlaced-Masked Fusion Network (GSIFN), and Perceiver IO—on agricultural time-series data. MDFCL builds individual graphs for each modality and applies unsupervised contrastive losses to align the embeddings of nodes, inducing cross-modal robustness (achieving 97.66% accuracy with image-only inputs). GSIFN develops an interlaced masking joint Transformer, capturing higher-order interactions with efficiency, along with self-supervised LSTM-based side tasks to counter redundancy (achieving 100.00% classification accuracy in rigorous 5-fold cross-validation). Perceiver IO adopts an implicit-latent bottleneck Transformer, providing heterogeneous streams of agricultural data flexibly with near-linear complexity without individual encoders per modality. Perceiver IO also achieved a similar result as compared to MDFCL (achieving 97.66% accuracy) while it has fewer parameters and requires less computations. Although these models have been found to be successful with generic multimodal tasks, their applicability to fusion of agro-sensor and growth-image data has thus far remained relatively less explored. We evaluate their adaptation versatility, advantages, and limitations in this study, providing actionable recommendations on fusion of agricultural image and time-series data to support precision agriculture with robustness, scalability, and efficiency.