Using wearable devices to monitor physical activity, particularly step count, is not a new concept. Many wearable fitness trackers also have a heart rate monitor and we’ve previously talked about the accuracy of that data. And if you have one of these devices, you may have noticed that many track energy expenditure, or how many calories you’re burning.
Shcherbina et al from Stanford University developed a wearable sensor evaluation framework to test the accuracy of heart rate and energy expenditure estimates from seven commercially available wearable devices. The Apple Watch, Basis Peak, Fitbit Surge, Microsoft Band, Mio Alpha 2, PulseOn, and Samsung Gear S2 were tested against the clinical gold standards, continuous telemetry and indirect calorimetry, while sitting, walking, running, and cycling. The 60 volunteers included in the study were of diverse backgrounds. The framework, protocol, and results are publicly accessible, and their three main conclusions are summarized below.
First, the tested devices can produce heart rate data with acceptable error for walking, running, and cycling. The lowest error rates were observed for cycling (median 1.8%, 95% CI 0.9%–2.7%) and the highest error rates were observed for walking (median 5.5%, 95% CI 3.9%–7.1%). Second, none of the tested devices could produce energy expenditure data with less than 20% error. Third, the Apple Watch had the best overall error profile, while the Samsung Gear S2 had the worst. The Apple Watch had the lowest error in heart rate, 2.0% (95% CI 1.2%–2.8%), while the Samsung Gear S2 had the highest heart rate error, 6.8% (95% CI 4.6%–9.0%).
This study also revealed that activity intensity and sex were significant predictors of heart rate measurement errors, where the error rate for males was significantly higher than that for females (P=0.0000456, effect size=0.044, z=3.505) across all devices.
These results generally replicate some of the prior work we’ve seen in heart rate measurement validation. It’s the energy expenditure measurements, which were way off the mark, that are the big take away from this study. To arrive at whatever guesstimate of energy expenditure these devices offer, innumerable assumption must be made to come up with some figure. And as it turns out, that really doesn’t work.