A Study on User Satisfaction regarding the Clinical Decision Support System (CDSS) for Medication

Article information

Healthc Inform Res. 2012;18(1):35-43
Publication date (electronic) : 2012 March 31
doi : https://doi.org/10.4258/hir.2012.18.1.35
1Graduate School of Public Health, Yonsei University, Seoul, Korea.
2The Catholic University of Korea School of Medicine, Seoul, Korea.
3Research Institute of National Rehabilitation Center, Seoul, Korea.
4Pusan National University School of Medicine, Yangsan, Korea.
5Korea Health and Welfare Information Service, Seoul, Korea.
Corresponding Author: Young Moon Chae, PhD. Graduate School of Public Health, Yonsei University, 134, Shinchon-dong, Sudaemoon-gu, Seoul 120-752, Korea. Tel: +82-2-2228-1524, Fax: +82-2-392-7734, ymchae@yuhs.ac
Received 2012 March 12; Revised 2012 March 22; Accepted 2012 March 23.

Abstract

Objectives

Many medication errors can occur when ordering and dispensing medicine in hospitals. The clinical decision support system (CDSS) is widely used in an effort to reduce medication errors. This study focused on the evaluation of user satisfaction with the CDSS for medication at a university hospital. Specifically, this study aimed to identify the factors influencing user satisfaction and to examine user requirements in order to further improve user satisfaction and drug safety.

Methods

The study was based on survey data from 218 users (103 doctors, 103 nurses, and 15 pharmacists) at a university hospital that uses the CDSS. In order to identify the factors influencing user satisfaction with the CDSS, a multiple linear regression was performed. In order to compare the satisfaction level among the professional groups, an analysis of variance (ANOVA) was performed.

Results

The reliability of information, decision supporting capability, and departmental support were significant factors in influencing user satisfaction. In addition, nurses were the most satisfied group, followed by pharmacists and doctors according to the ANOVA. Areas for further improvement in enhancing drug safety were real time information searching and decision supporting capabilities to prevent adverse drug events (ADE) in a timely manner.

Conclusions

We found that the CDSS users were generally satisfied with the system and that it complements the nationwide drug utilization review (DUR) system in reducing ADE. Further CDSS evaluation in other hospitals is needed to improve user satisfaction and drug safety.

I. Introduction

During the last half century, in the development and the diversification of medical field scientific technology, hospital information systems (HIS) has provided new knowledge needed for work safety and precision using IT, and has thus received attention as an important tool for enhancing the quality of health and medical services [1].

The rapid proliferation and development of the HIS has increased the standards of quality improvement of medical service and the optimization of hospital management. For the enhancement of quality and effectiveness of work process related to medical use, clinical decision support systems (CDSS) linked to computerized provider order entry (CPOE) system has come into use. However, compared to the emergence of electronic medical record (EMR), CDSS is still in an early stage as its use is insufficient, and its development and utilization for the prescription and dispensing of medicines are limited.

Through many studies, CPOE integrated with CDSS has reduced adverse drug reaction cases and medication errors, and this suggests that drug use safety has been enhanced [2-5]. In Korea, several studies have analyzed prescription changes before and after the application of CDSS for medication. Lee et al. [6] found that there was a reduction in medication errors after using CDSS for medication. However, for it to be effective in reducing medication errors, key users of CDSS need to be satisfied with it. Accordingly, user satisfaction is a critical success factor (CSF) for CDSS in many studies [2,4,7,8]. DeLone and McLean [9,10] subdivided success measures of information systems into six distinct categories: 1) system quality, 2) information quality, 3) user satisfaction, 4) usage, 5) individual impact, and 6) organizational impact. Van Der Meijden et al. [11] examined the determinants of success of inpatient clinical information systems according to the DeLone and McLean [9,10] framework. Park et al. [12] have analyzed the performance of the CDSS for medication in 38 hospitals by using the DeLone and McLean [9,10] success model of the information system and found that both system quality and information quality significantly influenced user satisfaction. However, these findings were based on only responses from pharmacists, excluding doctors who are responsible for maintaining drug safety.

This study aimed to investigate satisfaction among all users (doctors, nurses, and pharmacists) with CDSS and its effects on enhancing drug safety. Specifically, this study identified the factors influencing user satisfaction in terms of system satisfaction and information satisfaction, compared degree of user satisfaction among professional groups, and examined the areas for further improvement in the CDSS.

II. Methods

1. Subjects

Satisfaction of CDSS use for medication was investigated through a survey involving doctors, nurses, and pharmacists who worked in Pusan National University Hospital which has 1,170 beds. The survey was distributed directly during a visit to the hospital from September 7 through September 27, 2011. Two hundred and eighteen respondents were collected based on convenient sample and analyzed in the study. The study hospital did not require mandatory use of EMR for doctors and therefore most of doctors in the sample were young doctors.

2. Model and Statistical Analysis

Based on the DeLone and McLean [9,10] framework, we evaluated the user satisfaction of the CDSS by using three success measures (system quality, information quality, and support factor) as independent variables, and sum of two satisfaction measures (system satisfaction and information satisfaction) as dependent variables for a multiple regression analysis (Figure 1). Four measures were used for evaluating system quality: ease of system use, ease of understanding results, ease of understanding terminology, and usefulness of system user interface (UI). Five measures were used for evaluating information quality: information accuracy, timeliness, reliability, up-to-datedness, and decision support. In addition, three supporting factors were used for evaluating organizational supports: user training support, top management support, and related department support. All measures were originally coded in 5-point scale but converted into 3-point scale during the analyses in order to reduce empty cells.

Figure 1

Framework for the evaluation of clinical decision support system. UI: user interface, ANOVA: analysis of variance.

In addition, associations between three success measures and system overall satisfaction was analyzed based on cross-tabulation. We also analyzed the associations between three success measures and drug safety improvement. However, chi-square test was not carried out because of small cell size in many items. In addition, each of satisfaction measures was compared for three professional groups by using analysis of variance (ANOVA). The SAS 9.1 package (SAS Institute Inc., Cary, NC, USA) was used for statistical analysis of the data.

III. Results

1. Characteristics of the Subjects

The general characteristics of the survey respondents are shown in Table 1. The total number of respondents in the survey was 218; 34% were males and 66% were females. For age group distribution, 211 of them were under 40 years, comprising the largest age group. The occupational composition was 47% (doctors), 46% (nurses), and 7% (pharmacists). The respondents were mostly from the internal medicine and surgery departments, 42% and 40%, respectively. Finally, 59.3% of the respondents had 2-5 years of hospital work experience.

Table 1

General characteristics of the respondents

2. The Effect of Success Measures on User Satisfaction

Among the success measures, information reliability, decision supporting factor, and related department support factor were significant factors influencing user satisfaction as shown in Table 2. The R-square of this model was 0.65.

Table 2

The effects of success measures on user satisfaction

3. Analysis of Association between Success Measures and User Satisfaction

To analyze the association between success measures and user satisfaction in detail, we selected overall system satisfaction from system satisfaction measures and drug satisfaction improvement from information satisfaction measures.

1) Association between success measures and overall system satisfaction

As seen in Table 3, positive responses (Yes) in overall system satisfaction had consistently higher positive responses (Yes) in all success measures. Specifically, the highest positive response was information reliability (82.9%), followed by usefulness of system user interface (81.6%) and information timeliness (77.6%) in order. Compared with system quality and information quality measures, however, all support factors had lower percentage of positive satisfaction responses. The highest positive response on support factors was the related department support (48.0%), followed by user training support (44.8%).

Table 3

Association between success measures and overall system satisfaction

2) Association between success measures and drug safety improvement

As seen in Table 4, positive responses (Yes) in drug safety improvement also had consistently higher positive responses (Yes) in all success measures. While overall trends were similar between overall system satisfaction and drug safety improvement, orders of importance were different for two measures. Specifically, the highest positive response was decision supporting (82.0%), followed by usefulness of system user interface (75.3%) and information reliability (73.0%) in order. Compared with system quality and information quality measures, however, all support factors also had lower percentage of positive satisfaction responses. The highest positive response on support factors was user training support (43.8%), followed by the related department support (43.2%).

Table 4

The association between quality measures and improvement in drug safety

4. Comparison of Satisfaction by Professional Group

There are differences in user satisfaction by professional group among doctors, nurses, and pharmacists, who use the CDSS (Table 5). Nurses had the highest level of satisfaction, followed by pharmacists and doctors. Doctors and nurses had the highest satisfaction level in drug safety improvement, and pharmacists had the highest satisfaction level in information usefulness. When each factor of satisfaction level was compared, doctors were less satisfied in overall system satisfaction, task satisfaction, and information usefulness factors than nurses and pharmacists.

Table 5

Comparison of satisfaction by professional group

5. Comparison of Success Factors Influencing Drug Safety Improvement

We allowed for multiple responses to compare the responses on success factors influencing drug safety improvement. One hundred fifty-one (69.3%) respondents stated that the timely provision of drug information (product information, drug identification information, drug ingredient information, medication guide, national health insurance guide, etc.) was the function with the most support in improving drug safety (Table 6). Fifty-nine (27.1%) respondents said that the decision support function of CDSS (dose alert, drug interactions alert, duplication alert, and drug allergy alert) helped to improve drug safety, 35 (16.1%) respondents answered that medication support tools (medication instruction output tool, identification report output tool, administration tool) helped in drug safety improvement, and 5 (2.3%) respondents answered that checking function for drug utilization review (DUR) requirements by Health Insurance Review Agency (HIRA) also helped drug safety improvement.

Table 6

Comparison of success factors influencing drug safety improvement by professional group

Regardless of their profession, all respondents stated that drug information providing function provided the most support in drug safety improvement. The second most answered function was decision support function, of which 26.6% of doctors and 25.3% of nurses responded that it helped to improve drug safety. For the pharmacist group in which the decision support function does not apply, the second chosen function to drug safety improvement was medication support tools.

IV. Discussion

This study examined the effects of quality measures of CDSS (system quality and information quality) and its support factors (top management, department support, and user training) on satisfaction factors (user satisfaction and information satisfaction) by using the information system success model of DeLone and McLean [9,10]. We found that two information quality factors (information reliability and decision supporting capability) and one supporting factor (departmental support) were significant factors influencing user satisfaction.

When user satisfaction was separately analyzed by system satisfaction and information satisfaction, ease of use and decision support capability significantly influenced both user satisfaction factors. Ease of use was also a significant factor influencing user satisfaction in studies by Park et al. [12] and Kulkarni et al. [13]. This finding supported the ultimate goal of CDSS for medication, which provides information to help make a better decision in prescribing medicine.

We also analyzed the association between success measures and overall system quality and found that positive responses (Yes) in overall system satisfaction had consistently higher positive responses (Yes) in all success measures. Compared with system quality and information quality measures, however, all support factors had lower percentage of positive satisfaction responses. This shows an importance of management supports in improving user satisfaction suggested by the previous studies [9-11].

Degree of user satisfaction was the highest for nurses, followed by pharmacists and doctors. Low satisfaction level for doctors may be explained by the fact that the CDSS was primarily designed from administrative perspective focusing on reducing medication errors rather than doctors' clinical decision-aid such clinical practice guideline for prescription. Primary concern in information satisfaction also varied among professions. Drug safety was the highest for doctors and nurses, while usefulness of information was the highest for pharmacists. In addition, 69.3% of users responded that timely provision of drug information were the most important system functions to the CDSS in improving drug safety.

In conclusion, we found that user satisfaction with the capability of enhancing drug safety was reasonably high. However, real time safety checking function, which was a major concern for doctors, should be improved further in order to increase the satisfaction level of doctors and reduce medication errors. In addition, Our study showed that the dimensions of success defined by DeLone and McLean [9,10] for information management systems are applicable to the CDSS for medication, similar to a study on inpatient care information systems by Van Der Meijden et al. [11].

This study has a couple of limitations. First, the user satisfaction, which is the key indicator of CDSS evaluation, is a measurement of the perception on the system of users. Thus, there is a possibility of the indicator being subjective. Many studies have mentioned that quantifying measured factors is difficult when evaluating the outcomes of information systems. It is difficult to separate the effect of the information system objectively. However, presenting objective data, such as the cost-benefit analysis of information system [4], the real usage of detailed function of information system, and reduced time of decision-making, is needed.

Second, our findings may have external validity limitations. This study was conducted in only one hospital; as such, it is difficult to apply these results to other hospitals that may have different task process and characteristics. In addition, the study targeted one CDSS, so it is hard to generalize the results to user satisfaction on all CDSS. The environments of hospitals, such as task characteristics and size, are different from one another and they can be affected by exogenous variables deriving from circumstances of each hospital. For the results to be more applicable, systematic research of various hospital environments and systems will be needed.

Acknowledgments

Authors express sincere gratitude for the administrave support of the Pusan National University Hospital. This study is partially funded by the grant from the Korea Health Promotion Foundation.

Notes

No potential conflict of interest relevant to this article was reported.

References

1. Vegoda PR. Introduction to hospital information systems. Int J Clin Monit Comput 1987;4:105–109. 3585130.
2. Kaushal R, Shojania KG, Bates DW. Effects of computerized physician order entry and clinical decision support systems on medication safety: a systematic review. Arch Intern Med 2003;163:1409–1416. 12824090.
3. Chaudhry B, Wang J, Wu S, Maglione M, Mojica W, Roth E, Morton SC, Shekelle PG. Systematic review: impact of health information technology on quality, efficiency, and costs of medical care. Ann Intern Med 2006;144:742–752. 16702590.
4. Wolfstadt JI, Gurwitz JH, Field TS, Lee M, Kalkar S, Wu W, Rochon PA. The effect of computerized physician order entry with clinical decision support on the rates of adverse drug events: a systematic review. J Gen Intern Med 2008;23:451–458. 18373144.
5. Shamliyan TA, Duval S, Du J, Kane RL. Just what the doctor ordered: review of the evidence of the impact of computerized physician order entry system on medication errors. Health Serv Res 2008;43(1 Pt 1):32–53. 18211517.
6. Lee YT, Bae MY, Park JH, Choi CK, Bae SB, Chae YM. Evaluation of the effects of the CDSS for drug prescription for hospitals. J Korean Soc Health Inf Health Stat 2007;32:89–98.
7. Baroudi JJ, Orlikowski WJ. A short-form measure of user information satisfaction: a psychometric evaluation and notes on use. J Manag Inf Syst 1988;4:44–59.
8. Kim SW, Kang HT. An exploratory study on the success factors of knowledge management systems: focused on the role of perceived feedback. Entrue J Inf Technol 2011;10:149–162.
9. DeLone WH, McLean ER. Information systems success: the quest for the dependent variable. Inf Syst Res 1992;3:60–95.
10. DeLone WH, McLean ER. The DeLone and McLean model of information systems success: a ten-year update. J Manag Inf Syst 2003;19:9–30.
11. Van Der Meijden MJ, Tange HJ, Troost J, Hasman A. Determinants of success of inpatient clinical information systems: a literature review. J Am Med Inform Assoc 2003;10:235–243. 12626373.
12. Park J, Chae YM, Lee YT, Cho K, Kim J, Lee BH. Evaluation of CDSS for drug prescriptions based on success measures. J Korean Soc Med Inform 2009;15:293–301.
13. Kulkarni U, Ravindran S, Freeze R. A knowledge management success model: theoretical development and empirical validation. J Manag Inf Syst 2007;23:309–347.

Article information Continued

Figure 1

Framework for the evaluation of clinical decision support system. UI: user interface, ANOVA: analysis of variance.

Table 1

General characteristics of the respondents

Table 1

Values are presented as number (%).

Table 2

The effects of success measures on user satisfaction

Table 2

R-square = 0.65, p-value < 0.0001.

Table 3

Association between success measures and overall system satisfaction

Table 3

Values are presented as number (%).

Table 4

The association between quality measures and improvement in drug safety

Table 4

Values are presented as number (%).

Table 5

Comparison of satisfaction by professional group

Table 5

SD: standard deviation.

Table 6

Comparison of success factors influencing drug safety improvement by professional group

Table 6

Values are presented as number (%), and multiple responses were allowed.

HIRA: Health Insurance Review and Assessment Service, DUR: Drug Utilization Review.