by: Perry W. Payne, Jr., MD, JD, MPP

A recent article published in the Journal of Medical Internet Research focused on the development of better standards for reporting randomized controlled trials (RCTs) that use web-based or mobile health interventions.

The authors of the study were building on a prior initiative to improve suboptimal reporting of RCTs called the “Consolidated Standards of Reporting Trials” (CONSORT) statement. The researchers thought that the reporting of web-based and mobile health interventions in RCTs raised unique issues.

As a result, the goal of this study was to modify the CONSORT statement and develop a novel checklist for researchers and journals in order to improve the reporting RCT web-based and mobile health interventions. This is an important step in the evolution of mHealth  because documenting the efficacy of mobile and electronic interventions is critical to wide adoption by payors and policy makers.

Developing the CONSORT-EHEALTH Checklist

Researchers used a multi-step process to develop the checklist, called CONSORT-EHEALTH (CONsolidatedStandards OfReporting Trials of Electronic and Mobile HEalthApplications and onLineTeleHealth). First, using the existing CONSORT guideline items, researchers generated potential items for the checklist. These items arose from literature searches related to their topic of interest, extracting items from published RCTs and related guidelines, and using referee reports of articles submitted to the Journal of Medical Internet Research to identify items that reviewers described as “missing” from submissions to the journal.

The researchers formed a group of CONSORT-EHEALTH contributors which included “researchers, funders, consumers, journal editors, and industry” according to their article. The researchers developed an initial questionnaire with the list of potential items. The researchers placed their new web/mobile health specific items under the major headings of the CONSORT statement and labeled them as “subitems.”

The revised questionnaire was shared with the group of contributors at a face-to-face meeting hosted by the International Society for Research on Internet Interventions on April 6 – 8, 2011. The researchers didn’t indicate the size of the contributor group, though. The contributors were asked to suggest missing items and rate the proposed items o a scale of 1 – 5 with 1 being “subitem, not at all important” and 5 being “essential.” The questionnaire was also published in the Journal of Medical Internet Research in March of 2011 and the readers of the journal and other stakeholders were asked to comment and rate the importance of the subitems.

The subitems that were viewed as essential by at least 50% of people reviewing the questionnaire were rated as essential. For subitems with less than 50% but still receiving ratings of 4 or 5, they rated these items as highly recommended. Any subitems receiving less than 50% of respondents answering a 4 or 5 were removed from the questionnaire.

Study Results

The researchers received 55 responses to the questionnaire that was developed initially. Most users agreed with the subitems that were included in the questionnaire. One subitem was eliminated. It focused on the way that researcher institutional affiliations were displayed on a website or mobile application.

Some of the key subitems added are as follows:

  • Detailed description and documentation of the intervention – the researchers state that replication of web or mobile interventions requires much more detail than typical RCTs
  • Digital preservation of the intervention – in order to reproduce the intervention, the code that is used for the web/mobile intervention should be available. Given that code may be proprietary, at a minimum screenshots of the intervention should be provided
  • Data on attrition (non-use) and use of the intervention – researchers point out that research participants usually have more control of a web/mobile intervention and therefore are more likely to withdraw from a study complicating the studies. The level of use of an intervention is also more complicated because there are different degrees of use for a web/mobile intervention, unlike a drug trial. Researchers discuss a variety of different ways that use can be described (logging in to site, using certain pages of a site or mobile application, etc.)
  • Findability – researchers suggest that authors of study articles use similar terms in their articles and abstracts in order to facilitate quicker retrieval of articles in databases like PubMed

The Journal of Medical Internet Research adopted the questionnaire for RCT’s submitted to its journal. The final questionnaire that is used for all researchers submitting RCTs to Journal of Medical Internet Research can be viewed here.

Next Steps

The researchers view the CONSORT-EHEALTH checklist as a first step and indicate it will be “a living document in an iterative and ongoing development process” in order to keep pace with the evolving landscape of mobile and web health interventions. Along with keeping the checklist current, the researchers developed a plan to expand the use of their approach by more journals. They indicate that reporting guidelines for mobile/web intervention RCTs are lacking and there is a clear need to promote checklists like the one they’ve developed in multiple countries.

According to the researchers, their plan involves developing “a body of evidence to support usage of the guideline we intend to evaluate, elaborate on, and further develop the CONSORT-EHEALTH checklist.”

Their plan includes:

  • Piloting the questionnaire with the Journal of Medical Internet Research and collecting feedback from researchers using it
  • Analyzing retrospectively a randomly selected group of web-based RCTs which will lead to a manuscript
  • Creating a website and interactive toolkit focused on the checklist
  • Developing a standing working group to oversee the updates of the checklist
  • Performing a systematic analysis of web intervention RCTs published after the checklist was published in order to evaluate its impact and identify potential new subitems
  • Creating a searchable database of web and mobile intervention RCTs

The researchers have provided their tools in an open access manner and encourage other journals and researchers to use it. One missing aspect of the researchers’ plan is a set of deadlines for when the goals will be achieved. The iMedicalApps team will keep an eye out for any future developments from these researchers and report them to you.

Full Text of the Research Article may be seen here