Scholarlink

<p>While Large Vision Language Models (LVLMs) exhibit remarkable capabilities, their visual modality introduces a critical attack surface that can bypass text only safety alignments. This paper evaluates the vulnerability of LLaVA-1.5 to targeted adversarial visual prompts designed to induce malicious compliance. Using a Projected Gradient Descent (PGD) attack on the MM-SafetyBench dataset, we evaluate 1000 samples across five high risk categories. To eliminate false positives caused by superficial compliance, we apply a rigorous metric that strictly demands sustained, direct compliance without late stage refusals. Our results demonstrate that imperceptible visual perturbations effectively hijack safety guardrails, achieving Attack Success Rates (ASR) of 95% to 100% across all categories at perturbation budgets of <InlineEquation ID="IEq1"> <EquationSource Format="TEX">\(\epsilon \ge 8/255\)</EquationSource> </InlineEquation>. Furthermore, analysis of the Modality Gap (<InlineEquation ID="IEq2"> <EquationSource Format="TEX">\(\Delta _{\text {MG}}\)</EquationSource> </InlineEquation>) reveals that adversarial visual embeddings overpower textual safety constraints, forcing a malicious multimodal alignment. These findings underscore the inadequacy of current unimodal safety fine tuning and highlight the urgent need for robust, multimodal specific defense mechanisms.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Quantifying modality imbalance and visual jailbreak robustness in LLaVA via projected gradient descent

  • 期刊论文

  • 出版模式：

    开放获取

  • 发表日期：

    2026年5月8日

查看全文
  • Saklain Abdullah,
  • Riad Hossain,
  • Mahfuzulhoq Chowdhury

摘要

While Large Vision Language Models (LVLMs) exhibit remarkable capabilities, their visual modality introduces a critical attack surface that can bypass text only safety alignments. This paper evaluates the vulnerability of LLaVA-1.5 to targeted adversarial visual prompts designed to induce malicious compliance. Using a Projected Gradient Descent (PGD) attack on the MM-SafetyBench dataset, we evaluate 1000 samples across five high risk categories. To eliminate false positives caused by superficial compliance, we apply a rigorous metric that strictly demands sustained, direct compliance without late stage refusals. Our results demonstrate that imperceptible visual perturbations effectively hijack safety guardrails, achieving Attack Success Rates (ASR) of 95% to 100% across all categories at perturbation budgets of \(\epsilon \ge 8/255\) . Furthermore, analysis of the Modality Gap ( \(\Delta _{\text {MG}}\) ) reveals that adversarial visual embeddings overpower textual safety constraints, forcing a malicious multimodal alignment. These findings underscore the inadequacy of current unimodal safety fine tuning and highlight the urgent need for robust, multimodal specific defense mechanisms.