Large vision-language models exhibit strong in-context learning (ICL) capabilities, yet when and why visual context helps multimodal ICL remains poorly understood.
Empirical studies show a puzzling dichotomy: models sometimes effectively leverage visual demonstrations, yet often neglect them entirely.
We Propose VIB-ICL
We propose VIB-ICL, an information-theoretic framework that resolves this dichotomy through the Information Bottleneck principle.
Introducing Cross-Modal Information Gain (CMIG)
We introduce the Cross-Modal Information Gain (CMIG), which quantifies the additional mutual information that visual context provides about the target beyond textual context.
Generalization Bound
We derive a generalization bound showing that multimodal ICL's excess risk over text-only ICL is governed by the CMIG, proving that multimodal ICL provably outperforms text-only ICL when visual information is non-redundant.
Visual Context Neglect
We further prove that visual context neglect, often viewed as a failure mode, is the Information Bottleneck-optimal solution when visual information is redundant, yielding a closed-form Attention Reallocation Principle that prescribes how visual attention weights should be adaptively adjusted.
Instantiating the Principle
We instantiate this principle in the VIB-ICL algorithm, which estimates CMIG via variational bounds and dynamically reallocates attention.
Experiments and Results
Experiments on five benchmarks demonstrate consistent improvements of up to 4.7% accuracy gains and 35% reduction in required demonstrations, validating our theoretical predictions.