A recent study published in the BMJ by Semigran et al., (2015) looked at popular symptom checker apps and their ability to diagnose the “right way”.
They looked at 23 symptom checking apps and utilized 45 standardized patient vignettes that were equally divided into three categories: emergent care required, non-emergent care reasonable, and self care reasonable.
The following were outcomes, measures and results from the study:
Main outcome measures for symptom checkers that provided a diagnosis, our main outcomes were whether the symptom checker listed the correct diagnosis first or within the first 20 potential diagnoses (n=770 standardized patient evaluations). For symptom checkers that provided a triage recommendation, our main outcomes were whether the symptom checker correctly recommended emergent care, non-emergent care, or self care (n=532 standardized patient evaluations).
Results The 23 symptom checkers provided the correct diagnosis first in 34% (95% confidence interval 31% to 37%) of standardized patient evaluations, listed the correct diagnosis within the top 20 diagnoses given in 58% (55% to 62%) of standardized patient evaluations, and provided the appropriate triage advice in 57% (52% to 61%) of standardized patient evaluations. Triage performance varied by urgency of condition, with appropriate triage advice provided in 80% (95% confidence interval 75% to 86%) of emergent cases, 55% (47% to 63%) of non-emergent cases, and 33% (26% to 40%) of self care cases (P<0.001). Performance on appropriate triage advice across the 23 individual symptom checkers ranged from 33% (95% confidence interval 19% to 48%) to 78% (64% to 91%) of standardized patient evaluations.
The overall conclusion that most in the media made from this is symptom checkers are woefully inaccurate when it comes to getting the correct diagnosis and they are risk averse to a fault — sending patients to the ER when they don’t necessarily need to go.
The more flashy headline is one that states how symptoms checking apps don’t replace physicians — you don’t need a study to show you that. In reality, what I took away from the study was when the researchers compared the apps to telephone triage calls.
If anyone has had to deal with calling their physician’s after hours triage line, it can take anywhere from a few minutes to more than an hour to get an appropriate recommendation. I’ve had patients show up to the ER several times because they couldn’t get a final answer from their primary care doctor’s office in time or showing up much later than they should have because they had to wait to get a final response on the phone line.
The study’s following discussion of utilizing these apps as possible replacements for telephone triage lines was the most interesting part:
The accuracy of telephone triage recommendations, as compared to in-person physician recommendations, ranged from 61% in a study of pediatric abdominal pain to 69% in a multicenter observational study.35 36 A recent study of NHS Symptom Checkers and NHS Direct’s telephone triage line found triage advice from both to be comparable.9 Given their similar clinical logic, triage performance, and their negligible operation costs, symptom checkers could potentially be a more cost effective way of providing triage advice than nurse-staffed phone lines.
The other key takeaway for me was the lack of using existing evidence based protocols.
Symptom checkers that used the Schmitt or Thompson nurse triage protocols were more likely to provide appropriate triage decisions than those that did not: 72% (95% confidence interval 60% to 84%) v 55% (50% to 59%), P=0.01
If you want to replicate a triage experience, apps should at least utilize resources that have already been shown to work.
Semigran, H., Linder, J., Gidengil, C., & Mehrotra, A. (2015). Evaluation of symptom checkers for self diagnosis and triage: Audit study. BMJ.