Thesis Defense: VirtualHome: Building Socially Intelligent Agents via Simulation
Host
Antonio Torralba
CSAIL
Abstract:
While remarkable progress has been made in building autonomous agents for complex tasks, these have typically been studied in isolated environments. To build agents that can be deployed in the real world, we need to study them in more human-centric settings, where they can interact with other humans and agents. Moreover, for agents to assist us effectively, they need to be able to operate in these environments while understanding our intentions and beliefs, and learn to coordinate with us effectively and safely.
This thesis investigates the development of such assistive agents through simulation environments. In the first part of the thesis we introduce VirtualHome, a multi-agent platform for simulating human activities in household environments, and describe a knowledge base of daily human activities that can be executed in the simulator. Then, we present agents that can perform different tasks in this environment following human descriptions or demonstrations of an activity. Finally, we study agents that can perform activities together with other humans in the environment. We propose a framework to simulate humans in the environment at scale and propose two challenges, Watch-And-Help and Online-Watch-And-Help, to benchmark the performance of different assistive agents and test their effectiveness in assisting real humans to perform activities in VirtualHome. We show that the developed agents can be helpful to real humans, and that testing them with simulated humans is indicative of their performance when assisting the real ones. VirtualHome, along with the proposed challenges, offer a way to develop and test agents that can assist humans in a safe and scalable manner, before deploying them into the real world.
Speaker:
Xavier Puig
Committee:
Antonio Torralba, Professor MIT (Advisor)
Sanja Fidler, Associate Professor University of Toronto
Phillip Isola, Associate Professor MIT
While remarkable progress has been made in building autonomous agents for complex tasks, these have typically been studied in isolated environments. To build agents that can be deployed in the real world, we need to study them in more human-centric settings, where they can interact with other humans and agents. Moreover, for agents to assist us effectively, they need to be able to operate in these environments while understanding our intentions and beliefs, and learn to coordinate with us effectively and safely.
This thesis investigates the development of such assistive agents through simulation environments. In the first part of the thesis we introduce VirtualHome, a multi-agent platform for simulating human activities in household environments, and describe a knowledge base of daily human activities that can be executed in the simulator. Then, we present agents that can perform different tasks in this environment following human descriptions or demonstrations of an activity. Finally, we study agents that can perform activities together with other humans in the environment. We propose a framework to simulate humans in the environment at scale and propose two challenges, Watch-And-Help and Online-Watch-And-Help, to benchmark the performance of different assistive agents and test their effectiveness in assisting real humans to perform activities in VirtualHome. We show that the developed agents can be helpful to real humans, and that testing them with simulated humans is indicative of their performance when assisting the real ones. VirtualHome, along with the proposed challenges, offer a way to develop and test agents that can assist humans in a safe and scalable manner, before deploying them into the real world.
Speaker:
Xavier Puig
Committee:
Antonio Torralba, Professor MIT (Advisor)
Sanja Fidler, Associate Professor University of Toronto
Phillip Isola, Associate Professor MIT