Scholarlink

We present a three-stage pipeline for automated transportation infrastructure monitoring: GPS-based geofence creation, YOLOv11-powered detection, and Vision Language Model (VLM) ensemble assessment. Our calibrated ensemble employs isotonic regression for probability calibration and differential evolution optimization with population size 100, achieving 84.38% F1 score on graffiti detection with an optimized decision threshold of 0.715. The ensemble combines seven VLMs including cloud-based models (GPT-4o, GPT-4.1, Claude-3.7, GPT-o4 mini) and locally deployable models (LLaVA-1.6, Gemma-3, R1), which through iterative pruning was reduced to four models while maintaining performance. To address extreme data scarcity (3 real examples), we developed parametric synthetic generation using YOLO-World and SAM, expanding to 3,805 balanced samples across five colors (black, silver, white, red, blue). Experiments covered 227.58 mi across 778 unique bus stops in Reno, capturing 331,045 images with 15,389 labeled infrastructure elements. Geofence-triggered processing reduces computational requirements by 95%, processing only 5% of captured frames while enabling practical VLM deployment. Color-specific analysis revealed VLMs’ superior generalization compared to CNNs, which showed severe performance degradation on excluded colors—MobileNet achieving only 0.09 F1 on white graffiti when trained without white/silver examples.

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Enhancing Infrastructure Monitoring with Calibrated Vision Language Model Ensembles: A Graffiti Detection Case Study

  • 会议论文

  • 发表日期：

    2026年

查看全文
  • Gaetano Evangelista,
  • Houston Lucas,
  • Richard Kelley

摘要

We present a three-stage pipeline for automated transportation infrastructure monitoring: GPS-based geofence creation, YOLOv11-powered detection, and Vision Language Model (VLM) ensemble assessment. Our calibrated ensemble employs isotonic regression for probability calibration and differential evolution optimization with population size 100, achieving 84.38% F1 score on graffiti detection with an optimized decision threshold of 0.715. The ensemble combines seven VLMs including cloud-based models (GPT-4o, GPT-4.1, Claude-3.7, GPT-o4 mini) and locally deployable models (LLaVA-1.6, Gemma-3, R1), which through iterative pruning was reduced to four models while maintaining performance. To address extreme data scarcity (3 real examples), we developed parametric synthetic generation using YOLO-World and SAM, expanding to 3,805 balanced samples across five colors (black, silver, white, red, blue). Experiments covered 227.58 mi across 778 unique bus stops in Reno, capturing 331,045 images with 15,389 labeled infrastructure elements. Geofence-triggered processing reduces computational requirements by 95%, processing only 5% of captured frames while enabling practical VLM deployment. Color-specific analysis revealed VLMs’ superior generalization compared to CNNs, which showed severe performance degradation on excluded colors—MobileNet achieving only 0.09 F1 on white graffiti when trained without white/silver examples.