ML Tea: Collapse-Proof Non-Contrastive Self-Supervised Learning / Data Attribution in High Dimensions and without Strong Convexity

Speakers: Emanuele Sansone and Ittai Rubinstein

Bios:

Emanuele Sansone is a Postdoctoral Fellow jointly affiliated with MIT (CSAIL) and KU Leuven (ESAT). His research interests lie at the intersection between unsupervised learning and mathematical logic. His research ambition is to empower machines with the capability to acquire and discover knowledge from data in an autonomous manner. He was recently awarded the Marie Curie Global Fellowship for the program titled “Discovering the World through Unsupervised Statistical Relational Learning”.

Ittai Rubinstein is a third-year PhD student in Computer Science at MIT, advised by Sam Hopkins, supported by the Mathworks EECS Fellowship. His research centers on algorithms, with a focus on data attribution and robust machine learning. Before MIT, he led a research team at Qedma working on quantum error suppression and mitigation. He holds a master’s degree in computer science from Tel Aviv University, and a bachelor’s degree in mathematics, physics, and computer science from the Technion.

Abstracts:

Self-supervised learning (SSL) has unlocked the ability to learn general-purpose representations from vast amounts of unlabeled data. Despite its successes, significant challenges remain, limiting the applicability and democratization of SSL. One key challenge lies in the failure modes that arise during SSL training. In this talk, we distill essential principles for reliably avoiding these known collapses. We introduce a principled yet simplified design of the projector and loss function for non-contrastive SSL, grounded in hyperdimensional computing. Theoretically, we show that this design induces an inductive bias that naturally encourages representations to become both decorrelated and clustered, without explicitly enforcing these properties. This bias provably improves generalization and is sufficient to prevent common training failures, including representation, dimensional, cluster, and intracluster collapses. We further validate our theoretical insights on image datasets, showing that our approach produces representations that retain richer information about the observed data while avoiding memorization. This opens the door to learning more structured representations.

Data attribution estimates the effect of removing a set of samples from a model's training set without retraining the model from scratch and are used for interpretability, credit assignment, privacy and more. However, key approaches to data attribution significantly underestimate removal effects in the high-dimensional regime (#params >= Omega(#samples)), and existing theoretical analyses require strong convexity assumptions that rarely hold in practice, even for simple linear probes. In this talk, we will present a correction to the leading approaches to data attribution that improve accuracy in the high-dimensional regime and present the first theoretical guarantees for the accuracy of data attribution without strong convexity.