Your browser does not support the NLM PubReader view.

Go to this page to see a list of supporting browsers.

Evaluation of Mobile Health Applications Developed by a Tertiary Hospital as a Tool for Quality Improvement Breakthrough

Article information

Healthc Inform Res. 2015;21(4):299-306
Publication date (electronic) : 2015 October 31
doi : https://doi.org/10.4258/hir.2015.21.4.299
Yura Lee, MD, MS1,2,*, Soo-Yong Shin, PhD1,3,4,*, Ji-Young Kim3,4, Jeong Hun Kim, MS3,4, Dong-Woo Seo, MD, PhD4,5, Segyeong Joo, PhD3,6, Joong-Yeol Park, MD, PhD3,7, Woo Sung Kim, MD, PhD1,4,8, Jae-Ho Lee, MD, PhD1,3,5, David W. Bates, MD, MS9
1Department of Biomedical Informatics, Asan Medical Center, University of Ulsan College of Medicine, Seoul, Korea.
2Department of Breast and Endocrine Surgery, Asan Medical Center, University of Ulsan College of Medicine, Seoul, Korea.
3Ubiquitous Health Center, Asan Medical Center, University of Ulsan College of Medicine, Seoul, Korea.
4Medical Information Office, Asan Medical Center, University of Ulsan College of Medicine, Seoul, Korea.
5Department of Emergency Medicine, Asan Medical Center, University of Ulsan College of Medicine, Seoul, Korea.
6Department of Biomedical Engineering, Asan Medical Center, University of Ulsan College of Medicine, Seoul, Korea.
7Department of Endocrinology/Metabolism, Asan Medical Center, University of Ulsan College of Medicine, Seoul, Korea.
8Department of Pulmonary and Critical Care Medicine, Asan Medical Center, University of Ulsan College of Medicine, Seoul, Korea.
9Division of General Medicine and Primary Care, Department of Medicine, Brigham and Women's Hospital, Boston, MA, USA.
Corresponding Author: Jae-Ho Lee, MD, PhD. Department of Emergency Medicine, Asan Medical Center, University of Ulsan College of Medicine, 88 Olympic-ro 43-gil, Songpa-gu, Seoul 05505, Korea. Tel: +82-2-3010-3350, Fax: +82-2-2045-4126, rufiji@gmail.com
*These two authors contributed equally to this work.
Received 2015 August 12; Revised 2015 September 07; Accepted 2015 September 07.

Abstract

Objectives

To evaluate the mobile health applications (apps) developed by a single tertiary hospital in Korea with a particular focus on quality and patient safety.

Methods

Twenty-three mobile health apps developed by Asan Medical Center were selected for analysis after exclusion of the apps without any relationship with healthcare or clinical workflow, the apps for individual usage, and the mobile Web apps. Two clinical informaticians independently evaluated the apps with respect to the six aims for quality improvement suggested by the United States Institute of Medicine. All discrepancies were resolved after discussion by the two reviewers. The six aims observed in the apps were reviewed and compared by target users.

Results

Eleven apps targeted patients, the other 12 were designed for healthcare providers. Among the apps for patients, one app also had functions for healthcare providers. 'My cancer diary' and 'My chart in my hand' apps matched all the six aims. Of the six aims, Timeliness was the most frequently observed (20 apps), and Equity was the least observed (6 apps). Timeliness (10/11 vs. 10/12) and Patient safety (10/11 vs. 9/12) were frequently observed in both groups. In the apps for patients, Patient-centeredness (10/11 vs. 2/12) and Equity (6/11 vs. 0/12) were more frequent but Efficiency (5/11 vs. 10/12) was less frequent.

Conclusions

Most of the six aims were observed in the apps, but the extent of coverage varied. Further studies, evaluating the extent to which they improve quality are needed.

Keywords: Mobile Health; Telemedicine; Quality Improvement; Patient Safety; Patient-Centered Care

I. Introduction

Ubiquitous health (u-health) aims to provide innovative healthcare services that are consumer-oriented, customized, and accessible anytime, anywhere by ubiquitous computing [1]. More than 10 years have passed since the concept of u-health was proposed in Korea [1]. Despite very high expectations, u-health still is in its very early stages, in part because its clinical effectiveness has not been clearly demonstrated [12]. It is also struggling in Korea in part because of the complex debates about telemedicine. However, electronic health (e-health), a concept that is similar to u-health, which utilizes information systems by applying them to existing healthcare practices to improve the productivity of the work of the health service provider [1], has been growing in Europe and the United States even though issues regarding its clinical effectiveness have not yet been resolved [34567]. The e-health market has grown rapidly, especially after the emergence of mobile health (m-health) in 2009 [34578910]. m-Health services using both wearable devices and smartphones are also expanding [111213].

m-Health provides innovative healthcare services by using the unique characteristics of the smartphone such as mobility, connectivity, accessibility, and the capacity to gather and transmit data [91011121314], so it is considered a subtype of u-health in Korea [12]. Diverse u-health devices, including wearable devices, can be connected to smartphones, and sometimes the smartphone itself can be used as a u-health device [1516]. m-Health services and trials using smartphone sensors, such as cameras, accelerometers, and pedometers, are frequent now [131718].

Despite this rapid growth, challenges remain which must be addressed if m-health is to result in the hoped-for changes [3812]. According to Dr. Joseph Kvedar, Director of the Center for Connected Health of Partners Healthcare, one in ten U.S. patients has used a health tracker, but the majority stop using it within six weeks, and for m-health applications (apps), 80% are abandoned after only two weeks [19]. Several studies have suggested key features that encourage users to be more likely to continuously use such apps, but most trackers and apps have little impact, and more research is needed [192021]. m-Health must also be connected to personal health records or Electronic Health Records to provide a continuous care model and to improve patient outcomes [8], but such connectivity and interoperability remain major issues. The Apple HealthKit and ResearchKit have the potential to be breakthrough technologies in this area [22]. Another big challenge is determining the clinical effectiveness of m-health. Whether the use of m-health can improve patient outcomes must be determined [2356823]. Like telemedicine cases, not all m-health services can achieve this goal [324]. The common features of m-health that impact its outcomes must be researched [8]. Therefore, welldesigned clinical trials and meta-analyses are required to definitively determine the effectiveness of m-health [368]. Healthcare organizations in Korea have also developed many m-health apps and have provided m-health services [925]. However, few outcomes research studies have been done in this area. The six aims for quality improvement of the Institute of Medicine (IOM) in the United States can be applied to evaluating the benefits of m-health apps [26]. In 2001 the IOM recommended that "health care should be supported by systems that are carefully and consciously designed to produce care that is safe, effective, patient-centered, timely, efficient, and equitable" and "health information technology (HIT) must play a critical role to achieve the six aims [26]." Some studies reviewed HIT applications with respect to the six aims to evaluate their benefits and impacts [2728].

Asan Medical Center in Seoul, Korea, has developed more than 20 m-health apps since 2010. Because no other hospital in Korea has developed as many m-health apps as our hospital has, we evaluated the m-health apps developed by our hospital with a particular focus on the IOM's six aims with the intention of sharing our findings.

II. Methods

The information about the launch date, the date of the final update, and installed data for the apps was collected from the mobile apps server or hospital information system server with cooperation of the Ubiquitous Health Center in the hospital. We collected the information about the apps from the day of distribution of the app until December 31, 2014. The Institutional Review Board approved our study protocol.

1. App Selection and Categorization

Mobile apps developed by Asan Medical Center or in which there was a leading role of the hospital in the initial request were reviewed retrospectively. After excluding the apps with no relationship to healthcare or clinical workflow and the apps designed for individual usage, 23 apps were chosen for the final analysis. These 23 apps were categorized based on their target users (i.e., healthcare providers or patients) and their range of distribution (i.e., open to public or closed for inhospital use) (Figure 1). We divided the apps into two groups, according to the target user of the apps. In addition, the functions of the apps and the implementations of the six aims of the apps in each group were evaluated and compared.

Figure 1

Categorization of the 23 apps selected for analysis. The x-axis represents the target users, and the y-axis represents the range of distribution (Emergency in ordinary life1: emergency in ordinary life for patients, emergency in ordinary life2: emergency in ordinary life for clinicians).

2. The Six Aims for Quality Improvement

The functions and purposes of the apps were reviewed based on the six aims. The six aims selected by the IOM include the following:

1) Patient-centeredness

Providing care that is respectful of and responsive to individuals (i.e., apps for providing medical/health information to patients themselves, providing patients' authority to decide),

2) Effectiveness

Providing services based on scientific knowledge to all who could benefit, and refraining from providing services to those not likely to benefit (i.e., apps for verifying clinical guidelines and clinical information with evidence, apps for practice based on evidence, apps for clinical trials to make evidence),

3) Patient safety

Avoiding injuries to patients from the care that is intended to help them (i.e. apps for prevention, monitoring, and early detection of harmful events in the field of healthcare),

4) Timeliness

Reducing waits and sometimes harmful delays for both those who receive and those who give care (i.e., apps for providing information without limitations of place and/or time, bedside access to health information),

5) Efficiency

Avoiding waste, including waste of equipment, supplies, ideas, and energy (i.e., apps for fast and precise transmission of health information in clinical workflow to achieve good decision-making, apps for reducing cost, unnecessary processes and manpower),

6) Equity

Providing care that does not vary in quality because of personal characteristics, such as gender, ethnicity, geographic location, and socioeconomic status (i.e., apps for vulnerable population; the disabled, patients with chronic diseases, patients with mental diseases, pediatric patients, maternity patients, and the elderly).

3. Evaluation Questionnaires for the Apps

Using questionnaires, the 23 apps were reviewed to assess their operating system, target user, necessity of log-in, and user supporting method. These questionnaires, and the categories they included, were prepared by two clinicians who specialize in clinical informatics. Both clinicians designed the questionnaires and independently reviewed them. After the forms were filled out, the two reviewers discussed the results of each app and resolved all discrepancies.

III. Results

Table 1 presents the overall characteristics of the selected 23 apps. Eleven of the 23 apps were designed for use by patients and the general population, 12 targeted healthcare providers as users. Among the apps for patients, one app also had functions for healthcare providers. Figure 2 lists the number of users of each app. As expected, the public apps had more users.

Table 1

Characteristics of the 23 mobile apps analyzed, which were developed or released by Asan Medical Center (AMC)

Figure 2

Accumulated count of users from the day of distribution to December 31, 2014.

The frequency that the 23 apps involved the six aims is described in Table 2. 'My chart in my hand' and 'My cancer diary' were matched for all the six aims. The specific functions or purposes of the apps correlate with the six aims as follows:

Table 2

Compatibility of the mobile apps developed at Asan Medical Center with the six aims for quality improvement

1) Patient-centeredness

'My chart in my hand' and 'Pediatric cancer diary' provide medical data to patients and promote self-management by recording and analyzing personal health data (i.e., body weight, blood sugar level, and blood pressure level). 'Pharm consult' provides e-mail-based consultation for proper drug use and management of drug side effects. 'mAMIS' can help physicians show and explain the patient's information at the patient's bedside.

2) Effectiveness

Apps such as 'ASAN-in', 'Toxicology', and 'Emergency in ordinary life' provide evidence-based information to clinicians and/or patients. 'Pit-a-Pat', 'Quality of life', and 'Injini' are apps for clinical studies to build evidence.

3) Patient safety

'Wild plants in Korea' and 'My medications' are apps to prevent hazards encountered in daily life, and 'Ora-Quick' and 'Blood culture' prevent hazards in the clinical field. 'mAMIS', 'CPCR drugs', and 'Pediatric cancer diary' assist in the early detection and/or rapid management of hazardous events.

4) Timeliness

'm-AMIS' and 'My chart in my hand' provide access to patients' information independent of space and time. 'My medication' and 'My cancer diary' provide health information or support at the patient's bedside.

5) Efficiency

Apps such as 'AMC ARC', 'ASANnet', and 'ASAN talk' promote efficient networking and communication between healthcare providers. 'Emergency in ordinary life' and 'Patient transportation' help to create an efficient clinical workflow.

6) Equity

'Pediatric cancer diary', 'My chart in my hand', and 'Injini' provide care for vulnerable populations, such as cancer patients, pediatric patients, children with cognitive disorder or patients with chronic diseases.

The frequency of overlap with the functions of apps and the six aims present different dispositions according to the target user (Table 3) and the range of distribution (Figure 3). Timeliness (10 apps out of 11 patients-targeted apps vs. 10 out of 12 healthcare provider-targeted apps) and Patient safety (10/11 vs. 9/12) were frequently observed in both groups and evenly distributed. The remaining aims showed differences between patient-targeted apps and healthcare providertargeted apps. In the apps for patients, Patient-centeredness (10/11 vs. 2/12) and Equity (6/11 vs. 0/12) were more frequent, but Efficiency (5/11 vs. 10/12) was less frequent. All the apps matched with Equity, and all the apps except two that matched with Patient-centeredness were for patients. In contrast, the apps corresponding to Efficiency were predominantly those for healthcare providers. There was no Equity-matched app for healthcare providers.

Table 3

Apps that have functions or purposes that are compatible with the six aims according to the target user

Figure 3

Numbers of applications that have functions or purposes that are compatible with the six aims according to the target user.

IV. Discussion

It was clear that many apps for healthcare address a number of the key quality aims. For example, Timeless was predominant due to the nature of smartphones, which offer accessibility and mobility. Desktop-based programs has a lot of limitations in terms of timeless. Patient safety was well implemented in both groups. This implies that m-health can be beneficial in the quality aims. Most patient-targeted apps covered Equity and Patient-centeredness. Most apps for healthcare providers satisfied Efficiency. Interestingly, there was no app for healthcare providers that satisfies all of the six aims, in contrast with the apps for patients, such as 'My cancer diary' and 'My chart in my hand.' In addition, Equity was not matched for any app for healthcare providers. This reminds us to be aware of vulnerable populations when developing an app, even for healthcare providers [29].

There were some limitations of this study. First, the questionnaires used to assess the apps were evaluated by two clinicians. Since one of the clinicians participated in the development of the apps, any matches between the apps and the six aims might be biased. Also, the other clinician had limited ability to evaluate some apps which ended their services. Therefore, further evaluation of apps is necessary, focusing on currently in-service apps. For more objective and precise evaluation of healthcare-related apps in the future, the assessment of the six aims should be performed by experts uninvolved in their development and should encompass patients who are currently using the apps. Second, we found a disparity in the number of users in accordance with the range of distribution and/or the date of launch. In addition, the number of users who installed an app does not necessarily correlate with the level of usage. Since the extent of usage or the activity of apps is difficult to evaluate precisely, more advanced measures should be developed.

In this study, most of the apps designed for patients provide health information or patient medical information and satisfy Patient-centeredness. These apps are also a good tool for gathering health information from patients. Unfortunately, the user data collected in this way have not been actively applied to clinical use until now. However, the patients' generated data in clinical apps, with the merits of Timeliness and Effectiveness, would promote user interaction. In addition, tailored treatments acquired through user interactions with these healthcare apps would contribute to better Patient-centeredness and Patient safety. Therefore, the evaluation of an individual app is essential for efficient management and continual improvement of the app.

In conclusion, although this study was performed using the apps developed by a single tertiary hospital in Korea, the evaluation results demonstrate the enormous potential of mhealth apps as important tools to achieve breakthrough of quality improvement. To achieve this breakthrough, there should be much more study to improve m-health apps.

Acknowledgments

This research was supported by the Basic Science Research Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Science, ICT & Future Planning (NRF-2012R1A1A1014529).

Notes

Conflict of Interest: No potential conflict of interest relevant to this article was reported.

References

1. Lee Y, Chang H. Ubiquitous health in Korea: progress, barriers, and prospects. Healthc Inform Res 2012;18(4):242–251. 23346474.
2. Shin SY, Lee JH. Mobile health: a new breakthrough for u-health. Telecommun Rev 2013;23(3):8.
3. Silva BM, Rodrigues JJ, de la Torre Diez I, Lopez-Coronado M, Saleem K. Mobile-health: a review of current state in 2015. J Biomed Inform 2015;56:265–272. 26071682.
4. Ricciardi L, Mostashari F, Murphy J, Daniel JG, Siminerio EP. A national action plan to support consumer engagement via e-health. Health Aff (Millwood) 2013;32(2):376–384. 23381531.
5. Aitken M, Gauntlett C. Patient apps for improved healthcare: from novelty to mainstream Parsippany (NJ): IMS Institute for Healthcare Informatics; 2013.
6. Free C, Phillips G, Watson L, Galli L, Felix L, Edwards P, et al. The effectiveness of mobile-health technologies to improve health care service delivery processes: a systematic review and meta-analysis. PLoS Med 2013;10(1):e1001363. 23458994.
7. World Health Organization. mHealth: new horizons for health through mobile technologies Geneva, Switzerland: World Health Organization; 2011.
8. Powell AC, Landman AB, Bates DW. In search of a few good apps. JAMA 2014;311(18):1851–1852. 24664278.
9. Park JY, Lee G, Shin SY, Kim JH, Han HW, Kwon TW, et al. Lessons learned from the development of health applications in a tertiary hospital. Telemed J E Health 2014;20(3):215–222. 23909863.
10. Terry M. Medical Apps for Smartphones. Telemed J E Health 2010;16(1):17–22. 20070172.
11. Case MA, Burwick HA, Volpp KG, Patel MS. Accuracy of smartphone applications and wearable devices for tracking physical activity data. JAMA 2015;313(6):625–626. 25668268.
12. Miguez-Burbano MJ, Ergon E. Use of wearable monitoring devices to change health behavior. JAMA 2015;313(18):1865. 25965242.
13. Park YR, Lee Y, Lee G, Lee JH, Shin SY. Smartphone applications with sensors used in a tertiary hospitalcurrent status and future challenges. Sensors (Basel) 2015;15(5):9854–9869. 25923933.
14. Lee J. Smart health: concepts and status of ubiquitous health with smartphone. In : Proceedings of 2011 International Conference on ICT Convergence (ICTC); 2011 Sep 28-30; Seoul, Korea. p. 388–389.
15. Muhlestein JB. QTC intervals can be assessed with the AliveCor heart monitor in patients on dofetilide for atrial fibrillation. J Electrocardiol 2015;48(1):10–11. 25465798.
16. Ceja E, Osmani V, Mayora O. Automatic stress detection in working environments from smartphones' accelerometer data: a first step. IEEE J Biomed Health Inform 2015;6. 16. 10.1109/JBHI.2015.2446195. [Epub].
17. Naslund JA, Aschbrenner KA, Barre LK, Bartels SJ. Feasibility of popular m-health technologies for activity tracking among individuals with serious mental illness. Telemed J E Health 2015;21(3):213–216. 25536190.
18. Zvornicanin E, Zvornicanin J, Hadziefendic B. The Use of Smart phones in Ophthalmology. Acta Inform Med 2014;22(3):206–209. 25132717.
19. Infantino J. Patient engagement, interoperability needed for mobile health to succeed [Internet] Oakland (CA): iHealthBeat.com; 2014. cited at 2015 Aug 3. Available from: http://www.ihealthbeat.org/insight/2014/patient-engagement-interoperability-needed-for-mobile-health-to-succeed.
20. Yuan S, Ma W, Kanthawala S, Peng W. Keep using my health apps: discover users' perception of health and fitness apps with the UTAUT2 model. Telemed J E Health 2015;21(9):735–741. 25919238.
21. Bidmon S, Terlutter R, Rottl J. What explains usage of mobile physician-rating apps? Results from a web-based questionnaire. J Med Internet Res 2014;16(6):e148. 24918859.
22. Ritter S. Apple's research kit development framework for iPhone apps enables innovative approaches to medical research data collection. J Clin Trials 2015;5(2):e120.
23. Tomlinson M, Rotheram-Borus MJ, Swartz L, Tsai AC. Scaling up mHealth: where is the evidence? PLoS Med 2013;10(2):e1001382. 23424286.
24. Mistry H. Systematic review of studies of the costeffectiveness of telemedicine and telecare: changes in the economic evidence over twenty years. J Telemed Telecare 2012;18(1):1–6. 22101609.
25. Jang D, Shin SY, Seo DW, Joo S, Huh SJ. A smartphone-based system for the automated management of pointof-care test results in hospitals. Telemed J E Health 2015;21(4):301–305. 25654664.
26. Institute of Medicine, Committee on Quality of Health Care in America. Crossing the quality chasm: a new health system for the 21st century Washington (DC): National Academy Press; 2001.
27. Chaudhry B, Wang J, Wu S, Maglione M, Mojica W, Roth E, Morton SC, Shekelle PG. Systematic review: impact of health information technology on quality, efficiency, and costs of medical care. Ann Intern Med 2006;144(10):742–752. 16702590.
28. Restuccia JD, Cohen AB, Horwitt JN, Shwartz M. Hospital implementation of health information technology and quality of care: are they related? BMC Med Inform Decis Mak 2012;12:109. 23016699.
29. Lee J. The impact of health information technology on disparity of process of care. Int J Equity Health 2015;14(1):34. 25889891.

Article information Continued

© 2015 The Korean Society of Medical Informatics

This is an Open Access article distributed under the terms of the Creative Commons Attribution Non-Commercial License (http://creativecommons.org/licenses/by-nc/4.0/) which permits unrestricted non-commercial use, distribution, and reproduction in any medium, provided the original work is properly cited.

Funded by : National Research Foundation of Korea
Award ID : NRF-2012R1A1A1014529

Figure 1

Figure 1

Categorization of the 23 apps selected for analysis. The x-axis represents the target users, and the y-axis represents the range of distribution (Emergency in ordinary life1: emergency in ordinary life for patients, emergency in ordinary life2: emergency in ordinary life for clinicians).

Figure 2

Figure 2

Accumulated count of users from the day of distribution to December 31, 2014.

Figure 3

Figure 3

Numbers of applications that have functions or purposes that are compatible with the six aims according to the target user.

Table 1

Characteristics of the 23 mobile apps analyzed, which were developed or released by Asan Medical Center (AMC)

Table 1

OS: operating system, PHR: personal health record, OPD: outpatient department, EMR: Electronic Medical Record, HIV: human immunodeficiency virus.

aFinal update: the date of latest version, checked on June 28, 2015. bUser count: accumulated count of users, assembled from the day of distribution to December 31, 2014. cEmergency in ordinary life1: emergency in ordinary life for patients. Emergency in ordinary life2: emergency in ordinary life for clinicians.

Table 2

Compatibility of the mobile apps developed at Asan Medical Center with the six aims for quality improvement

Table 2

Emergency in ordinary life1: emergency in ordinary life for patients, Emergency in ordinary life2: emergency in ordinary life for clinicians

Table 3

Apps that have functions or purposes that are compatible with the six aims according to the target user

Table 3

Values are presented as number (%).