Luo, Y. (CSE) – Rethinking Contrastive Learning: From Fundamental Limits to Capability Transfer Across Foundation Models

Contrastive learning has become one of the central recipes of modern AI. By learning from comparisons, telling which examples belong together and which do not, it produces the embeddings behind search, retrieval, and the multimodal systems that connect foundation models to the world. Its success rests on assumptions that are often taken for granted: that the contrastive objective can be optimized efficiently, that an embedding of modest dimension can faithfully capture the comparisons, and that a new capability must be trained into an embedding model through more contrastive data. My research revisits each of these assumptions. My completed work shows that the first two can fail at a fundamental level: even finding a locally optimal embedding can be intractable, and an embedding with too few dimensions can do no better than a trivial one. Rethinking the third, new abilities need not come from more contrastive training: we show that an embedding model can borrow some of an LLM’s instruction-following ability without any training. Going forward, we propose to extend this perspective to foundation models at large, with implications for lowering the cost of building and updating AI systems, and connections to model merging, knowledge distillation, multimodal alignment, and interpretability.
Event Host: Yiyuan Luo, PhD Student, Computer Science & Engineering
Advisor: Vaggos Chatziafratis
Zoom: https://ucsc.zoom.us/j/99506265345?pwd=YabUybEHGfZdtXZam8bXekqaTtz4B9.1
Passcode: 443041