With a variety of sensors that can be used to monitor one’s health and fitness, consumer electronics are becoming increasingly sophisticated. We introduced sleep sensing in the Nest Hub a few years ago. This feature makes use of radar technology known as Soli to analyze sleep patterns[1] when the device is placed close to the bed. More recently, we demonstrated that the Soli radar platform’s fully contactless frequency modulated continuous wave (FMCW) radar technology can monitor vital signs like heart rate and breathing rate during sleep and meditation. In “UWB Radar-based heart rate monitoring: A transfer learning approach,” we present new research demonstrating that radar-based heart rate measurement can be performed with ultra-wideband (UWB) technology, which is already found in a lot of mobile phones. Despite its widespread use for features like secure vehicle unlocking and precise item location, UWB’s potential for radar sensing has largely been overlooked. We demonstrate how this existing hardware can be leveraged for vital sign monitoring, such as measuring heart rate (HR).
Radar sensors available on consumer devices
Millimeter wave frequency-modulated continuous wave (mm-wave FMCW) and impulse-radio ultra-wideband (IR-UWB) radar systems are the ones that have shown the most promise for measuring vital signs from consumer devices. FMCW technology was used by Google in the past to detect sleep, movement, and gestures on the Soli radar platform. This meant that we already had a lot of data, research, and machine learning algorithms that had been trained for these tasks, like monitoring heart rate with FMCW radar. In the meantime, radar capabilities are also provided by UWB, a multipurpose technology that has gained popularity and is becoming increasingly available on a growing number of current mobile phone models and other consumer devices. The radar capabilities of UWB have been thus far largely untapped, with current UWB applications leaning more on non-radar uses like localization and tracking, vehicle unlock features, or data transfer.
overcoming the obstacle presented by touchless sensing It is difficult to use radar to detect HR without making any contact because the much larger movements caused by breathing and general body motion easily obscure the minute movements of the chest wall caused by the heartbeat. The distinctive nature of the radar signal comes into play at this point. Its three-dimensional spatial resolution uses distance and direction to focus its measurement. Because of this, the radar can precisely define a “measurement zone” all the way around a person’s torso. As a result, it can isolate reflections coming from the chest area while ignoring stationary background objects or movements occurring outside this zone. It simultaneously samples the signal at a rate of up to 200Hz, which is fast enough to capture the heartbeat’s subtle and rapid motion. We developed a new method that makes optimal use of these unique 2-dimensional spatio-temporal properties of the radar signal to achieve highly accurate heart rate measurement.
bridging the gap between different kinds of radar We looked into whether we could apply the features learned from FMCW radar, where we had access to extensive previous studies and datasets, to UWB radar. The two radar systems operate using completely different physical principles. Mm-wave FMCW transmits a continuous sinusoidal wave whose frequency increases linearly with time, periodically sweeping a frequency range, while UWB transmits very short pulses with duration on the order of a few hundred picoseconds to a few nanoseconds. For the first time, our research demonstrates that learned features can be transferred between different kinds of radar for measuring vital signs. Because of its high potential for use and level of difficulty, we decided to start with heart rate.
Developing a new deep learning model for heart rate from radar
To accomplish this task, we developed a novel deep learning framework designed to model the complex spatial-temporal relationships in radar signals for HR estimation. To begin processing the input data, the architecture makes use of a 2D ResNet, one axis of which represents time and the other the spatial measurements. This initial stage is designed to extract features from the fine-grained spatio-temporal patterns created by chest wall movements.
The model uses average pooling to collapse the spatial dimension after this step. A 1D ResNet, which is designed to analyze the signal solely along the temporal dimension, receives the resulting feature set. From the features that were extracted in the first stage, this second stage identifies the longer-term, periodic patterns that are characteristic of a heartbeat. For heart rate measurement, the model achieves a mean absolute error (MAE) of 0.85 when trained with our FMCW dataset. The previous error rate was cut in half by this finding, which is a significant improvement over previous state-of-the-art results on this dataset.
Transferring learned features to ultra-wideband radar
Using a setup that placed the UWB radar sensor in positions where users typically hold their phones, such as on a table in front of them or on their lap, we then conducted a study that collected UWB radar data in addition to electrocardiogram (ECG) and photoplethysmogram (PPG) data as our ground truth for heart rate. The UWB radar dataset was much smaller than the FMCW dataset, which contained 980 hours of data, with 37.3 hours. The range resolution of the UWB radar dataset was significantly lower than that of the FMCW dataset because its bandwidth was significantly lower and its configuration was similar to that of a mobile phone. To ensure that our model was optimized to transfer to the UWB dataset, we retrained it after performing additional pre-processing steps to modify the mm-wave FMCW radar data to better resemble the target IR-UWB data, effectively lowering its range resolution. We then fine-tuned this model on the IR-UWB dataset, achieving an MAE of 4.1 bpm and mean absolute percentage error (MAPE) of 6.3%, a 25% reduction over the baseline error rate. By selecting the best model that had been trained from scratch on our UWB dataset, we were able to achieve 5.4 bpm MAE and 8.4% MAPE as our baseline for performance on UWB radar. We were able to make it possible for the UWB radar to meet the Consumer Technology Association’s requirements for consumer device heart rate measurement: an accuracy of up to 5 bpm MAE and 10% MAPE.
Ensuring accuracy in different scenarios
We looked at how well our model performed in each dataset’s various scenarios and user conditions to ensure its accuracy and dependability. In situations that were adequately represented, we discovered that performance on heart rate measurement is consistent for both kinds of radar. The FMCW radar, for instance, whose data were collected during overnight sleep sessions, maintains performance in all sleep positions and even when a person moves between positions. For UWB radar, both tested device positions relative to the user—on a table in front of them or in their lap—are equally accurate for measuring heart rate. See the full research paper for additional information on this subgroup analysis and other findings.
The big picture: Everyday health monitoring
The measurement of a person’s heart rate provides fundamental insight into their cardiovascular status and physiological responses to various health conditions, making it useful for a variety of health, fitness, and wellness applications. This demonstration of heart rate measurement could be a step towards using mobile devices to measure even more complex and subtle health signals from the heart and large blood vessels.
While wearable devices like fitness bands and rings have popularized continuous monitoring of health and fitness, the ability to measure heart rate in a contactless manner with consumer-device–grade radar sensors allows the benefits of this technology to reach a much wider audience of smartphone users. For this study, we focused on heart rate while sleeping (for FMCW) and on a setup where the radar sensor was in positions where the phone is usually held during use (UWB). As technology develops, continuous monitoring may become available in a variety of everyday settings and seamlessly integrate with a user’s daily activities. What this means for devices to come In light of the increasing use of ultra-wideband (UWB) technology in mobile phones, this work brings us one step closer to making contactless heart rate measurement possible on consumer devices. Although our study did not include direct testing using mobile phones in a real-world setting, this research establishes the crucial groundwork for such future applications.
A core finding of this work is the demonstration that a model trained on one type of radar (FMCW) can be successfully adapted for another (UWB) to measure heart rate. This transfer learning approach is a significant step forward. It suggests a more effective course of action for future research and development, one in which the fundamental information gleaned from massively existing datasets can be used to develop new devices. This method speeds up the time it takes to add such features to consumer devices by allowing for a more streamlined process rather than starting from scratch with extensive data collection for each new piece of hardware.
