Wang, Z. (CSE) – Scaling Robust Visual Representation Learning: From Convolutional Architectures to Vision-Language Models

Visual encoders learned at scale, from convolutional networks and Vision Transformers to CLIP, now serve as the perceptual front end of Vision-Language Models (VLMs).
Yet they remain brittle: their accuracy degrades under natural distribution shifts, and imperceptible adversarial perturbations can change their predictions or the text that a VLM generates.
Robustness has also lagged behind scale.
It is commonly attributed to the self-attention architecture of Transformers; its most reliable defense, adversarial training, has been too expensive to apply beyond small models and datasets; and VLMs have been defended only by post-hoc fine-tuning that sacrifices zero-shot capability.
In this dissertation, we show that robustness can be designed into architectures, scaled with efficient training, and built into vision-language models.
First, through controlled architecture studies under a fixed training recipe, we identify three Transformer-inspired designs—patchified inputs, large convolution kernels, and fewer activation and normalization layers—that let pure CNNs match or exceed the out-of-distribution robustness of Vision Transformers of similar size.
Second, we introduce AdvXL, an adversarial training framework that pre-trains at reduced token length with weak attacks, briefly fine-tunes at full resolution with strong attacks, and uses a frozen CLIP text encoder to learn from web-crawled image-text data.
Scaling to a one-billion-parameter model trained on more than one billion images sets new ImageNet-1K records under AutoAttack, improving the previous best $\ell_\infty$-, $\ell_2$-, and $\ell_1$-robust accuracy by 11.4\%, 14.2\%, and 12.9\%.
Third, we propose Double Visual Defense, which adversarially pre-trains CLIP from scratch on web-scale data and adds adversarial visual instruction tuning to LLaVA.
Evaluated on more than 20 datasets, the resulting models improve robustness over prior defenses by about 20\% in zero-shot ImageNet classification, about 30\% in image captioning, and about 20\% in visual question answering, while hallucinating less and staying close to non-robust models on clean benchmarks.
Together, these results show that robustness is not tied to a particular architecture and need not be traded for capability: efficient training at scale makes it possible to build robustness into visual foundation models from the start.
Event Host: Zeyu Wang, PhD Candidate, Computer Science & Engineering
Advisor: Cihang Xie
Zoom: https://ucsc.zoom.us/j/94378973643?pwd=GN3DqdWgUTmOfetlQ9xFgccjPQKzex.1
Passcode: 778543