Machine learning models can achieve strong test performance while relying on demographic or acquisition-related shortcuts.
We propose counterfactual (CF) marginalisation as a test-time evaluation procedure for assessing robustness of classification models to such variables.
Given a CF image generator, we intervene on nuisance parent variables such as age or sex, generate CF versions of each test image, and average predictions over a target intervention distribution.
This produces intervention-aware predictions that marginalise demographic effects while preserving patient-specific latent information.
We use these predictions to define metrics for CF risk, calibration, stability and worst-case sensitivity.
We demonstrate this framework's utility for quantitative robustness evaluation.