Securing Vision Transformers Against Adversarial Attacks via Input Purification in Trusted Execution Environments

Hardware-assisted defenses based on Trusted Execution Environments (TEEs) seek to protect vision transformers from adversarial examples by hiding and isolating gradient information. By running sensitive segments of the training or inference process within the TEE, these methods restrict an adversary’s ability to observe gradient signals and thus create an illusion of robustness under conventional evaluation settings. In this work, we show that such perceived robustness is circumventable. We introduce a novel adversarial attack, ShieldBreaker, which employs backward pass differentiable approximation (BPDA) to break such defenses by approximating the true gradients through their protected and nondifferentiable components. Our findings demonstrate that simply obscuring or limiting gradient access through hardware isolation does not yield true adversarial robustness. As a result, we propose an input purification defense designed to remove adversarial perturbations from the input. Our experiments show that our defense prevents up to 75% of adversarial attacks on average while maintaining model accuracy.