[2020-21, in contact with Philips Research] Contactless Vital Signs and Activity Monitoring via Sensor Fusion Camera (or Optical sensor) plus Radar

Cameras have been used to measure vital signs from human face and body without direct contact of skin, including heart rate (variability), respiration rate, blood oxygenation saturation (infrared), pulse transit time (multi-site measurement), skin temperature (thermography).
The use of cameras also enables the measurement of human behaviors/activities and high-level visual semantic/contextual information that assists health monitoring, such as facial expression analysis for pain/discomfort detection, body motion for bed exit/fall detection or sleep staging, activity recognition for actigraphy.

Radio Frequency (RF) based methods, such as Radar in its different implementations (Doppler/UWB/FMCW) have been proposed for health monitoring. They can obtain high monitoring accuracy for movement-based measurement such as respiration rate (from movement of the thorax and abdomen), activity recognition and fall detection (from movement of the whole body and its parts), posture estimation, leading to various health applications such as patient monitoring in care units, sleep monitoring, and in-home monitoring.

Comparing these two sensing modalities for health monitoring, each of them has its pros and cons. The limitations of cameras are mainly the privacy concerns (especially for home-based sleep monitoring) and measurement depth (sensitive to occlusions like bed blanket). The limitations of Radar are that it cannot measure blood perfusion as camera photoplethysmography (PPG) does, and that it is sensitive to the mutual position of the sensor with respect to the person monitored.

To make the most of these two sensing technologies, we propose in this Master Thesis project two assignments to explore the fusion of radar & camera for health monitoring, specifically to see whether the combination can (i) enrich the measured vital/body signs, (ii) increase measurement robustness and coverage. Note that to keep the advantage of radar in privacy, the “camera” used for fusion here is particularly a “low resolution camera” or “single-pixel camera” (e.g. single or multiple photo-detectors).

Assignment

1] Fusion between radar and optical sensor (single-pixel/low-resolution camera) for vital signs monitoring: Based on the existing camera-based vital signs monitoring framework of Philips Research, add a radar to investigate whether such a combination can improve the measurement accuracy/coverage, such as camera for PPG-related measurement (HR, HRV); radar and camera jointly for respiration and motion measurement. For example, the relevant case is sleep monitoring where the subject is covered by a thick blanket where camera cannot measure respiratory motion from the chest.
[2] Fusion between radar and optical sensor (single-pixel/low-resolution camera) for activity monitoring: Based on the existing radar-based activity recognition framework of TU Delft, add a camera/optical sensor (i.e. measure light reflections from human body) to investigate whether such a combination can improve the activity classification (e.g. joint training of multi-dimensional signals), while still remaining the privacy advantage of radar. For example, the relevant case is activity monitoring (e.g. fall detection) in privacy concerned use case like in bedroom or bathroom.

This initial investigation should aim at demonstrating feasibility in healthy subjects (“proof-of-concept”) in a simulated lab environment. Experiments on actual patients are beyond the scope of this assignment.
The student is expected to perform a combination of experimental and signal processing work, contributing to the design and validation of a setup to collect relevant radar and camera data, as well as the development of signal/image processing algorithms that can prove the concept.
The student is expected to spend the majority of his time at TU Delft using the facilities of the MS3 research group, but attending periodic meetings involving the team at Philips Research Eindhoven. A placement at the Philips facilities in Eindhoven can also be considered in due course, depending on progress and needs within the project workload.

Contact

dr. Francesco Fioranelli

Microwave Sensing, Signals and Systems Group

Department of Microelectronics

Last modified: 2021-01-19