Steal the Patch Size: Adversarially Manipulate Vision-Language Models
By Kai Hu, Akash Bharadwaj, Weichen Yu, Matt Fredrikson
This paper introduces a black-box model-stealing attack that recovers private vision-tokenizer configurations of deployed VLMs, including patch size and preprocessing, via a task-level side channel from ViT patchification. Aligning synthetic grid images with the hidden patch grid causes periodic accuracy drops that reveal the patch size.