ML Tea: Pandemic-Potential Viruses are a Blind Spot for Frontier Open-Source LLMs / Theoretical Guarantees for Learning with Unlabeled Data in Online Classification

Speakers: Laura Luebbert and Jonathan Schafer

Zoom password: 114091
Abstracts:

We study large language models (LLMs) for front-line, pre-diagnostic infectious-disease triage, a critically understudied stage in clinical interventions, public health, and biothreat containment. We focus specifically on the operational decision of classifying symptomatic cases as \emph{viral} vs. \emph{non-viral} at first clinical contact, a critical decision point for resource allocation, quarantine strategy, and antibiotic use. We create a benchmark dataset of first-encounter cases in collaboration with multiple healthcare clinics in Nigeria, capturing high-risk viral presentations in low-resource settings with limited data. Our evaluations across frontier open-source LLMs reveal that (1) LLMs underperform standard tabular models and (2) case summaries and Retrieval Augmented Generation yield only modest gains, suggesting that naïve information enrichment is insufficient in this setting. To address this, we demonstrate how models aligned with Group Relative Policy Optimization and a triage-oriented reward consistently improve baseline performance. Our results highlight persistent failure modes of general-purpose LLMs in pre-diagnostic triage and demonstrate how targeted reward-based alignment can help close this gap.

Practitioners commonly train classifiers using unlabeled data in addition to labeled data, because labeled data is often harder to obtain. However from a theoretical perspective, the question of whether and how unlabeled data can offer provable benefits in classification tasks is still not fully understood. In this talk, I discuss two recent works on the power of unlabeled data in online learning, a popular mathematical model of supervised learning. We show that (1) for some concept classes, access to unlabeled data can guarantee a quadratic reduction in the number of learner mistakes, and (2) in all cases the reduction can never be more than quadratic. This resolves a problem that remained open for 30 years.