We investigate language in different contexts: from how it is learned, to how it is grounded in visual perception, all the way to how machines can readily interact with humans.
To this end we utilize and integrate tools from various disciplines, including Natural Language Processing, Information Retrieval and Computer Vision. We have developed START, a general-purpose question answering system and parser, an early version of which inspired today's Q&A systems such as Siri and Google Assistant. As part of our quest to understand human language we model language learning, the development of language, and how knowledge is transferred between languages. To enable communication with machines, we investigate how perception interacts with language and how robots can be controlled.
We aim to learn language by distant supervision through captioned videos, similarly to how children learn language through interacting with the world around.
The goal of this project is to model the process of ‘full interpretation’ of object images, namely the ability to identify and localize all semantic features and parts that are recognized by human observers.
