For years researchers at MIT’s Computer Science and Artificial Intelligence Lab (CSAIL) have been working on a device that can measure a person’s movement and vital signs without any physical contact, by instead analyzing the wireless signals in their environment. Led by MIT professor Dina Katabi, the team has employed the device in nursing homes and hospital environments to remotely monitor the sick and the elderly for health issues that include Parkinson’s, dementia and, most recently, even covid-19.
This week, the team has shown that they’ve gone a step further: their system can now identify what activity a person is doing, whether that’s sleeping, reading, cooking or watching TV - and all without capturing any sensitive visual data of the activities.
Katabi says that the new system, which her team calls “RF-Diary,” allows caregivers to monitor vulnerable populations in a non-obtrusive, privacy-preserving way - whether that means care professionals observing their assisted-living residents, or siblings wanting to keep tabs on their elderly parents who are living alone.
RF-Diary uses a floor map of a person’s living space to be able to learn what sorts of actions and activities happen in different locations and with different objects throughout the space. The system was trained by recording wireless signals of volunteers performing a range of activities which are each labeled by human annotators using short captions. After that training process, the system is able to generalize to new people and homes that it’s never seen before.
Experiments showed that RF-Diary was able to generate accurate captions about a person’s activities even in dark and occluded settings where video-based approaches would normally fail. The system can achieve greater than 90 percent classification accuracy on more than 30 different household actions.
Katabi says that the team developed RF-Diary with privacy concerns top of mind. To set up the system, the person being monitored first has to agree to execute several sets of moves as a form of consenting to being monitored. The system then asks the user to walk around their specific living space - which eliminates the possibility of RF-Diary being used to monitor an area that the resident doesn’t have access to.
As a next step, the researchers plan to adapt the system to work in real-world homes and hospitals, with the goal of making it a commercial product that could be bought for anyone with health issues or limited mobility.
Katabi co-wrote a paper about RF-Diary with lead authors Lijie Fan and Tianhong Li, both PhD students at MIT CSAIL, alongside postdoctoral associate Yuan Yuan. It will be presented as an oral presentation this week at the European Conference on Computer Vision (ECCV). The authors will also present a real-time demo of RF-Diary at ECCV’s demo session.