Back to Events

[Thesis Defense] Steering Robots with Inference-Time Interactions

Speaker

Felix Yanwei Wang
EECS/CSAIL

Host

Julie Shah
AA

February 25 2025

12:00P - 1:30P

Location

45-792
(the big glass room in the middle on the 7th floor)
Virtual Event

Date: Tuesday, February 25, 2025

Time: 12:00 PM - 1:30 PM

Location: 45-792

Zoom: https://mit.zoom.us/j/95052951960

Abstract:

Imitation learning has driven the development of generalist policies capable of autonomously solving multiple tasks. However, when a pretrained policy makes errors during deployment, there are limited mechanisms for users to steer its behavior. While collecting additional data for fine-tuning can address such issues, doing so for each downstream use case is inefficient at scale. My research proposes an alternative perspective: framing policy errors as task misspecifications rather than skill deficiencies. By enabling users to specify tasks unambiguously via interactions at inference-time, the appropriate skill for a given context can be retrieved without fine-tuning. Specifically, I propose (1) inference-time steering, which leverages human interactions for single-step task specification, and (2) task and motion imitation, which uses symbolic plans for multi-step task specification. These frameworks correct misaligned policy predictions without requiring additional training, maximizing the utility of pretrained models while achieving inference-time user objectives.

Thesis Supervisor: Julie Shah

Committee Members: Leslie Kaelbling, Jacob Andreas, Dorsa Sadigh

Contact: felixw@mit.edu

Add to Calendar 2025-02-25 12:00:00 2025-02-25 13:30:00 America/New_York [Thesis Defense] Steering Robots with Inference-Time Interactions Date: Tuesday, February 25, 2025Time: 12:00 PM - 1:30 PMLocation: 45-792Zoom: https://mit.zoom.us/j/95052951960Abstract:Imitation learning has driven the development of generalist policies capable of autonomously solving multiple tasks. However, when a pretrained policy makes errors during deployment, there are limited mechanisms for users to steer its behavior. While collecting additional data for fine-tuning can address such issues, doing so for each downstream use case is inefficient at scale. My research proposes an alternative perspective: framing policy errors as task misspecifications rather than skill deficiencies. By enabling users to specify tasks unambiguously via interactions at inference-time, the appropriate skill for a given context can be retrieved without fine-tuning. Specifically, I propose (1) inference-time steering, which leverages human interactions for single-step task specification, and (2) task and motion imitation, which uses symbolic plans for multi-step task specification. These frameworks correct misaligned policy predictions without requiring additional training, maximizing the utility of pretrained models while achieving inference-time user objectives.Thesis Supervisor: Julie ShahCommittee Members: Leslie Kaelbling, Jacob Andreas, Dorsa SadighContact: felixw@mit.edu TBD

Part of

Related Events