EI Seminar - Jacob Steinhardt - Large Language Models as Statisticians

Given their complex behavior, diverse skills, and wide range of deployment scenarios, understanding large language models---and especially their failure modes---is important. Given that new models are released every few months, often with brand new capabilities, how can we achieve understanding that keeps pace with modern practice?

In this talk, I will present an approach to this that leverages the skills of language models themselves, and so scales up as models get better. Specifically, we leverage the skill of language models *as statisticians*. At inference time, language models can read and process significant amounts of information due to their large context windows, and use this to generate useful statistical hypotheses. We will showcase several systems built on this principle, which allow us to audit other models for failures, identify spurious cues in datasets, label the internal representations of models, and factorize corpora into human-interpetable concepts.

This is joint work with many collaborators and students, including Ruiqi Zhong, Erik Jones, and Yossi Gandelsman.