Healthc Inform Res Search

CLOSE


Healthc Inform Res > Volume 26(3); 2020 > Article
Choi, Kim, Kim, and Chang: Building Consensus on the Priority-Setting for National Policies in Health Information Technology: A Delphi Survey

Abstract

Objectives

With growing attention on the healthcare industry as a potential market for big data and artificial intelligence in the Fourth Industrial Revolution, countries around the world are introducing and developing various policies and projects related to health information technology (HIT). To assist prioritizing HIT topics in policy making, this study adopts the Delphi technique to garner expert opinions from various fields of health informatics.

Methods

Data were collected from November 2019 to February 2020 using the Delphi technique through two rounds of surveys administered via email. The Delphi panel consisted of 16 experts with a high level of experience in health informatics. They were from the Health Information Policy Advisory Committee of the Ministry of Health and Welfare of Korea, and the board of directors of the Korean Society of Medical Informatics. The experts were asked to assess the importance, urgency, and difficulty of HIT topics in three domains: technology, application, and infrastructure.

Results

Of the 40 topic items, a 100% agreement was reached for the importance of 6 items, including 2 items in technology, 1 item in application, and 3 items in infrastructure domains. Especially, Quadrant I of a 2×2 matrix showing high importance and high urgency included 7 items in the technology domain, 2 items in the application domain, and 13 items in the infrastructure domain.

Conclusions

Most items with high importance and urgency belonged to the infrastructure domain. The findings indicated that fostering an infrastructural environment should be polices with top priorities of HIT.

I. Introduction

With the increasing burden of healthcare expenditure due to population aging, and the high prevalence of chronic diseases worldwide in recent years, there is a growing emphasis on the importance of disease prevention and lifestyle management [1,2]. This phenomenon has contributed to advancing information technology (IT), and has shifted the paradigm of healthcare from diagnosis and treatment of a disease to prevention and management of the disease, and from hospital-centered care to consumer-centered care [3,4]. In other words, healthcare is transforming to adopt the P4 approach: predictive, preventive, personalized, and participatory healthcare [5].
The IT-based healthcare is being recognized as a solution for improving the quality of health services, increasing patients’ convenience, and extending healthy life expectancy, while lowering the national burden of healthcare costs [6,7]. Furthermore, with its high added value, and potential for creating jobs, the IT-based healthcare is expected to positively contribute to the overall economy, which has motivated countries worldwide to foster this innovation as a new growth engine, and cultivate an environment to achieve sustainable economic growth [3,7].
Since the early 2000s, developed countries have prepared and promoted national-level policies on health information. In 2012, the World Health Organization (WHO) and the International Telecommunications Union (ITU) published the National e-Health Strategy Toolkit to provide guidelines on the vision, action plans, monitoring, and assessment for eHealth, based on which countries have initiated their efforts to develop strategic and comprehensive policies to foster the new industry [8,9]. The European Commission (EC) organized the 15th eHealth Network meeting (eHN) in 2019, and its Multiannual Work Programme published four eHealth strategies to be accomplished between 2018 and 2021 [10]. The four strategies comprise empowering patients, innovative use of health data, enhancing continuity of care, and overcoming implementation challenges. By structuring medical clusters, the German government intervened in the intercorporate network to promote the timely implementation of technologies for health innovation, and sharing of information, which has contributed to advancing Germany’s healthcare technology and boosting profit [7].
In the United States, the Office of the National Coordinator for Health Information Technology (ONC) evaluated health information technology (HIT) accreditation organizations with increased transparency, reliability, and efficiency of the Electronic Health Record (EHR) process. Following the Health Information Technology for Economic and Clinical Health Act (HITECH) enacted in 2009, the government has supported physicians and hospitals in implementing EHR, and had provided incentives to healthcare providers who implemented it [11]. Based on this, the United States developed a large-scale cohort to collect data for precision medicine, and disclosed data through the Blue Button project. Canada also established a nonprofit organization called Health Info-way, and launched the ACCESS 2022 to help people access and use personal health information [12]. As described here, majority of the countries worldwide put efforts to ensure the clarity of regulations to advance the digital healthcare industry and to improve their regulations to enable efficient use of medical and health data acquired in healthcare [3].
South Korea has enormous potential in terms of its IT infrastructure, the National Health Insurance System, top-level workforce, and application of healthcare big data [13]. Korean government is striving to establish legal guidelines on digital healthcare-related technologies and information, as well as a regulatory system that will promote the development of HIT, lay a foundation for research, foster workforce, and expand healthcare services [3]. However, compared to other advanced countries, promotion of the digital healthcare industry is slow due to multiple regulations, disagreement among stakeholders, and ambiguous strategies which may substantially hinder effective investment and application of technologies in healthcare [9], highlighting the need for a discussion and consensus based on health information policies at the national level [3,6].
To promote sustainable policies on HIT, each country needs to devise systematic and complementary strategies tailored to their health status, education, and technology in collaboration with stakeholders, instead of blindly implementing the policy guidelines used by the developed countries [9]. Therefore, it is necessary to identify specific HIT policies based on Korean healthcare information strategies and garner expert opinions to develop a policy roadmap. Consequently, to provide a foundation for developing HIT policies and strategies in Korea, this study grouped topics of health information into technology, application, and infrastructure domains. And the HIT items included in each domain were evaluated using the Delphi technique, based on the importance, urgency, and difficulty.

II. Methods

1. Expert Panel for the Delphi Survey

Panel selection is a critical process in a Delphi study, as the panel experts’ experience and knowledge regarding the topic determines the quality of study findings [14,15]. The research team contacted 17 members of the Health Information Policy Advisory Committee of the Ministry of Health and Welfare of Korea, and 23 members of the board of directors of the Korean Society of Medical Informatics (KOSMI) as the panel for this study. Ludwig [16] recommended a panel size of 15–20, while Gordon [17] recommended 15–35; however, panel size may differ depending on the purpose and design of the study, and the minimum panel size required for the purpose of the study is recommended [18]. In this study, total of 16 experts participated; 5 experts from the Health Information Policy Advisory Committee of the Ministry of Health and Welfare of Korea and 11 from the KOSMI.

2. Survey Questionnaire

The draft of the Delphi questionnaire first developed with 45 items that were derived from review of reports from the WHO-ITU’s National e-Health Strategy Toolkit [9], the EC’s eHealth strategy [10], and papers suggesting the direction of policies related HIT in Korea [1,3,4,6,7,9]. The initial set of items were revised and added by the research team, which finally consisted of 47 items, divided into three domains—technology, application, and infrastructure.
The content validity of the questionnaire was assessed using a 4-point Likert scale (1 = not relevant, 2 = somewhat relevant, 3 = quite relevant, and 4 = very relevant) by 6 members of the board of directors of the KOSMI, who are experts in the health informatics field. They were also asked to provide comments for any necessary revision for items. Of 47 items, 22 items received the item-content validity index (I-CVI) of less than 0.8. Among 22 items, 12 items were deleted, 9 items were revised, and 1 item was added based on comments and the research team discussion.
The first round of the Delphi questionnaire consisted of 36 items comprising technology (14 items), application (6 items), and infrastructure (16 items) domains. In the second round of the survey, the questionnaire with 40 items was constructed after revising based on the response from the first round survey; 15 items for the technology domain (10 for Development, 3 for Standards, and 2 for Security); 6 items for the application domain (3 for Service Development and 3 for Data Utilization); 19 items for the infrastructure domain (5 for Law/Regulation, 3 for Professional Workforce, 4 for Building an Organizational Foundation, and 7 for Policy and Support). The experts were asked to rate each item for the importance, urgency, and difficulty of HIT policy making with a 4-point Likert scale (1 = strongly disagree, 2= disagree, 3 = agree, and 4 = strongly agree). The importance, urgency, and difficulty were operationally defined as how significant each item is, how fast it needs to be dealt with, and how hard it is to be completed.

3. Delphi Survey

The modified Delphi technique was used in this study that enables researchers to determine the number of survey rounds based on the objective and topic of the study [19]. However, researchers are required to devise strategies to reach a consensus among experts within a shortened number of rounds to save time and cost. To this end, researchers provided a structured questionnaire in the first round of survey to help panelists quickly focus on the research topic, and establish an expert group that helps in facilitating the survey process [18,20]. Therefore, two rounds of survey were conducted and a structured questionnaire was provided in the first round. The research team facilitated the survey and to discuss the composition of the items and survey procedure.
In the first round, the importance, urgency, and difficulty of tasks in technology, application, and infrastructure domains were rated with a structured questionnaire, and open-ended questions were included such that experts could present their opinions about the items. In the second round, a structured questionnaire was also used, which highlighted the revised or added items with individual responses and agreement rate (Number of panelists who chose 3 or 4/Total number of responses) in the first round. Experts were asked to re-rate or confirm their own first-round ratings to 36 items (Supplementary A).
The first and second rounds were administered on December 27, 2019 and February 17, 2020, respectively, via email. A reminder email was sent to the panelists whose responses were delayed. The second round of the survey was conducted on all 16 panelists who participated in the first round of the survey (100% response rate).

4. Data Analysis

Because the Delphi technique utilizes both closed- and open-ended questions, quantitative and qualitative data were collected and analyzed [15]. The responses to the open-ended questions in the first round were reflected in the second questionnaire after discussion with the expert group. The ratings of importance, urgency, and difficulty for each item in the first and second rounds of the Delphi survey were classified into “disagree” (ratings of 1 or 2) and “agree” (ratings of 3 or 4), and the agreement rate was computed by dividing the number of panelists who chose “agree” by the total number of panelists who responded. In a Delphi survey, a 50% or higher agreement is generally regarded as being reached a consensus [15,21], and in this study, it was decided that a consensus was reached when the agreement rate was 70% or higher.
The agreement rates for importance and urgency computed in the second round were analyzed using a 2×2 matrix by placing healthcare information policy items into four quadrants. The priorities of the items in Quadrant I, which have the greatest perceived importance and urgency, were established according to their difficulty level.

5. Ethical Considerations

This study was approved by the Institutional Review Board of Yonsei University (No. Y-2019-0141) where the first author belongs. Information about the purpose and procedure of the study was attached to the email sent to the study participants. The information specified that participation is completely voluntary and that participants have the freedom to withdraw from the study at any time.

III. Results

1. Results from the First Round of the Delphi Survey

Out of the initial 36 items of HIT topics, 29 items (81%) on importance, 23 items (64%) on urgency, and 15 items (42%) on difficulty were reached the consensus (70% or higher agreement). In addition, based on experts’ opinions about the items, 15 out of the 36 items were revised, and 4 new items were added, which made up a total of 40 items for the second round.

2. Results from the Second Round of the Delphi Survey for Each Domain

In the second round, consensus (≥70%) was reached on the importance of 32 out of 40 items. Tables 1,3 present the agreement rate for importance, urgency, and difficulty for each domain, and the quadrants in the importance-urgency matrix analysis. Of the 40 items, a 100% agreement was reached for the importance in 6 items, including 2 items in the technology domain, 1 item in the application domain, and 3 items in the infrastructure domain.
As shown in Table 1, in the technology domain, the items with 100% agreement on importance were Internet of Things (1.1.1), and Technology for information security (1.3.2), and an item with 100% agreement on urgency was Personal health record (1.1.7). The following 4 items had the next highest rate of agreement on importance: Artificial intelligence (1.1.2), Personal health record (1.1.7), Interoperability (1.2.2), and Personal information protection (1.3.1).
In the application domain, 100% agreement on importance and urgency was reached for Data science (2.2.2), and 100% agreement on urgency was reached for Data quality (2.2.1) (Table 2). In the infrastructure domain, 100% agreement on importance and urgency was reached for 2 items: Personal information protection and ethical management (3.1.3) in the Law/Regulation category, and Developing and implementing formal training curricula (3.2.3) in the Professional Workforce category (Table 3). The remaining items with 100% agreement on importance or urgency belonged to the Policy and Support category: there was 100% agreement on the importance of Assessing the appropriateness of HIT (3.4.5), and 100% agreement on the urgency of Managing EMR certification system and process (3.4.6), and Utilization of health HIT (3.4.7).

3. Integration of Importance, Urgency, and Difficulty

To examine the priorities of the items for HIT policies, a 2×2 matrix with importance and urgency as the axes was used to classify items with an 83% agreement rate on importance and 80% agreement on urgency (Figure 1). Seven items in the technology domain, 2 items in the application domain, and 13 items in the infrastructure domain were included in Quadrant I. Table 4 lists 22 items for HIT policies included in Quadrant I, which features high importance and urgency in the ranking of their difficulty. None of the items perceived to be important were included in Quadrant IV, which indicates relatively lower urgency, and policies included in Quadrants II and III had low perceived importance, and varying levels of perceived urgency.
The experts stated that among items in Quadrant I, Improving expertise of practitioners (3.2.2) in the Professional Workforce category and Linking R&D of HIT (3.4.1) in the Policy and Support category were relatively less difficult (Table 4). The items with the highest perceived difficulty were Personal information protection (1.3.1) and Technology for information security (1.3.2) in the Security category of the technology domain. Furthermore, as at least 75% agreement was reached on the importance of policies in Quadrant II, which have relatively less perceived importance and high urgency, some of these items could be considered in the determination of policy priorities, such as the Certification of HIT and devices (3.1.2) in the infrastructure domain and Services adopting technology (2.1.3) in the application domain.

IV. Discussion

The findings of this study were derived from health information experts via a Delphi survey to build consensus on the prioritization of HIT topics in the Korean context. Considering feasibility, items in Quadrant I, which featured high perceived importance and urgency with low difficulty, would be the policies that could be promptly implemented. The experts found 22 items to be the most important and urgent, and many of these belonged to the infrastructure domain. The highest agreement was shown on the importance and urgency of personal information protection, and ethical management in the Law/Regulation category of the infrastructure domain. There was also a strong consensus on the importance of technology for information security in the Security category under the technology domain.
In particular, the difficulty of personal information protection, and ethical management in the Law/Regulation category in the infrastructure domain was rated as moderate at 69%. Thus, this topic should be addressed as a priority in health information policy making. Establishment of a relevant legal and institutional environment is essential for the preparation of HIT policies and strategies. The United States and countries in Europe developed and promoted a relevant legal system to collect, exchange, and utilize data while securing accurate health information in order to vitalize service-oriented projects [22,23].
The results also show the importance of training and continuing education programs offered by professional and academic organizations. The education programs aim to improve the expertise of healthcare workforce as well as HIT professionals and practitioners by implementing the formal training curricula for HIT. Training the workforce in healthcare informatics has been emphasized in both developed and developing countries [2426]. For example, in 2006, the American Medical Informatics Association launched a 10×10 program in partnership with any academic or other organizations with the aim of training 10,000 informaticians by 2010 in the healthcare informatics area [26]. Emerging areas of health informatics are data analytics, data governance, privacy and security, and interoperability [27]. The national HIT strategy should not only focus on establishing the infrastructure, technology, and application from the planning stage, but also on the training and improvement of workforce and national capabilities to operate the healthcare system.
To promote the quality and standardization of the HIT, the ONC has provided for the Certification of HIT, such as EHR and devices [28], and the European Institute for Health Records (EuroRec) also provides quality labeling and certification for EHR [29]. Korea also established the Korea Health Information Service for EMR certification in 2019. Currently, 8 EMR vendors and healthcare organizations have received EMR certifications [30].
This study is significant in that consensus was drawn from leading health information experts in Korea, the members of the Health Information Policy Advisory Committee of the Ministry of Health and Welfare of Korea, and the board of directors of the KOSMI to develop and prioritize HIT policies and strategic planning. Although survey findings were derived from only 16 experts, we were able to reach consensus on the policies after two rounds using the Delphi method. However, the fact that the panel of experts only consisted of 16 experts is a limitation of this survey, and a larger scale study should be conducted to obtain opinions from more experts and the general public to build HIT policies and strategies.
In conclusion, this study shows how to build consensus on the priority-setting for national HIT policies using the Delphi survey with a panel of experts in HIT. According to the main finding of the survey, majority of items with high importance and urgency belonged to the infrastructure domain, indicating that the fostering of an infrastructural environment should be prioritized in national HIT policies. Korea’s strengths lie in its IT infrastructure, medical insurance systems for all citizens, advanced professionals, and utilization of health big data. With the widespread implementation of EMR, picture archiving and communication systems, and relevant advanced IT and software having top level importance worldwide, prioritized items and domains identified in this study should be paid attention and invested for preparing national HIT policies and strategies at present and in the near future.

Supplementary Information

Acknowledgments

This research was supported by the Ministry of Health and Welfare, Republic of Korea. Special thanks are extended to 16 experts in health information and technology who completed the Delphi survey for their time and efforts.

Notes

Conflict of Interest

Three authors, Mona Choi, Jung A Kim, and Hyejung Chang, are members of the Editorial Board of Healthcare Informatics Research. However, they did not involve in the peer review evaluation and decision process of this article. Otherwise, no potential conflict of interest relevant to this article was reported.

Supplementary Materials

Supplementary materials can be found via https://doi.org/10.4258/hir.2020.26.3.229.

Figure 1
Results of the second round of Delphi survey: Importance-Urgency Matrix.
hir-26-3-229f1.jpg
Table 1
Results of the second round of Delphi survey: technology domain (15 items)
Category Item Importance (%) Urgency (%) Difficulty (%) Quadranta
1.1 Development (1.1.1) IoT (Internet of Things) 100 93 75 I
(1.1.2) AI (artificial intelligence) 94 88 81 I
(1.1.3) Blockchain 56 53 73 III
(1.1.4) AR/VR (augmented/virtual reality) 63 44 44 III
(1.1.5) 3D printing 63 44 56 III
(1.1.6) Genome sequencing 81 69 81 III
(1.1.7) PHR (personal health record) 94 100 75 I
(1.1.8) Mobile or web application 63 63 13 III
(1.1.9) Cloud 75 56 50 III
(1.1.10) 5G 75 63 56 III

1.2 Standards (1.2.1) CDM (common data model) 63 44 31 III
(1.2.2) Interoperability 94 88 75 I
(1.2.3) Terminology standards 86 93 64 I

1.3 Security (1.3.1) Personal information protection 94 88 88 I
(1.3.2) Technology for information security 100 94 88 I

a Quadrant: See Figure 1.

Table 2
Results of the second round of Delphi survey: application domain (6 items)
Category Item Importance (%) Urgency (%) Difficulty (%) Quadranta
2.1 Service Development (2.1.1) Usage of health information 75 69 31 III
(2.1.2) Telemedicine 69 50 50 III
(2.1.3) Services adopting technology 81 88 56 II

2.2 Data Utilization (2.2.1) Data quality 93 100 81 I
(2.2.2) Data science 100 100 81 I
(2.2.3) Data platform for precision health 81 75 94 III

a Quadrant: See Figure 1.

Table 3
Results of the second round of Delphi survey: infrastructure domain (19 items)
Category Item Importance (%) Urgency (%) Difficulty (%) Quadranta
3.1 Law/Regulation (3.1.1) Approval of HIT and devices 88 94 75 I
(3.1.2) Certification of HIT and devices 81 94 56 II
(3.1.3) Personal information protection and ethical management 100 100 69 I
(3.1.4) Cost and fee systematization of medical services 75 81 81 II
(3.1.5) Standardization 94 94 67 I

3.2 Professional Workforce (3.2.1) Monitoring the supply and demand of professionals 88 87 69 I
(3.2.2) Improving expertise of practitioners 88 93 44 I
(3.2.3) Developing and implementing formal training curricula 100 100 75 I

3.3 Organizational Foundation (3.3.1) Establishing a public control tower 88 80 75 I
(3.3.2) Organizing a stakeholder advisory panel 94 87 69 I
(3.3.3) Building a channel to reflect opinions from society 94 93 63 I
(3.3.4) Bolstering the role and independence of professional organizations in HIT 80 81 69 II

3.4 Policy and Support (3.4.1) Linking R&D of HIT 88 88 50 I
(3.4.2) Supporting companies with HIT 56 38 25 III
(3.4.3) Prioritizing investment areas 63 69 38 III
(3.4.4) Reinforcing education on rights of personal information 80 80 47 II
(3.4.5) Assessing the appropriateness of HIT 100 94 69 I
(3.4.6) Managing EMR certification system and process 94 100 56 I
(3.4.7) Utilization of HIT 88 100 63 I

HIT: health information technology, R&D: research and development, EMR: Electronic Medical Record.

a Quadrant: See Figure 1.

Table 4
Ranks of easiness for items in Quadrant I of the importance-urgency matrix (22 items)
Ranka Difficulty (%) Item Importance (%) Urgency (%)
1 44 (3.2.2) Improving expertise of practitioners 88 93
2 50 (3.4.1) Linking R&D of HIT 88 88
3 56 (3.4.6) Managing EMR certification system and process 94 100
4 63 (3.3.3) Building a channel to reflect opinions from society 94 93
4 63 (3.4.7) Utilization of health information 88 100
6 64 (1.2.3) Terminology standards 86 93
7 67 (3.1.5) Standardization 94 94
8 69 (3.1.3) Personal information protection and ethical management 100 100
8 69 (3.4.5) Assessing the appropriateness of HIT 100 94
8 69 (3.3.2) Organizing a stakeholder advisory panel 94 87
8 69 (3.2.1) Monitoring the supply and demand of professionals 88 87
12 75 (1.1.1) IoT (Internet of Things) 100 93
12 75 (3.2.3) Developing and implementing formal training curricula 100 100
12 75 (1.1.7) PHR (Personal Health Record) 94 100
12 75 (1.2.2) Interoperability 94 88
12 75 (3.1.1) Approval of HIT and devices 88 94
12 75 (3.3.1) Establishing a public control tower 88 80
18 81 (2.2.2) Data science 100 100
18 81 (1.1.2) AI (Artificial intelligence) 94 88
18 81 (2.2.1) Data quality 93 100
21 88 (1.3.2) Technology for information security 100 94
21 88 (1.3.1) Personal information protection 94 88

R&D: research and development, HIT: health information technology, EMR: Electronic Medical Record.

a Sorted in ascending order of difficulty (for the same difficulty, items are sorted in descending order of importance).

References

1. Korea Ministry of Food and Drug Safety. Strategic report of smart healthcare devices and technology standards [Internet]. Cheongju, South Korea: Ministry of Food and Drug Safety; 2018 [cited at 2020 July 11]. Available from: https://www.nifds.go.kr/brd/m_18/down.do?brd_id=80&seq=12448&data_tp=A&file_seq=1.

2. Australian Government Department of Health. National e-Health Strategy [Internet]. Canberra, Australia: Department of Health; 2008 [cited at 2020 July 11]. Available from: https://www1.health.gov.au/internet/main/publishing.nsf/Content/National+Ehealth+Strategy.

3. Park JW, Shim WH, Lee JS. A study for Promoting digital healthcare in Korea through an improved regulatory system. Informatiz Policy 2018;25(1):60-81.

4. Song YJ. The 4th Industrial Revolution and digital healthcare policies [Internet]. Daejeon, South Korea: Institute of Information and Communications Technology Planning and Evaluation; 2018 [cited at 2020 July 11]. Available from: https://www.itfind.or.kr/WZIN/jugidong/1832/file3874377402637323357-183201.pdf.

5. Flores M, Glusman G, Brogaard K, Price ND, Hood L. P4 medicine: how systems medicine will transform the healthcare sector and society. Per Med 2013;10(6):565-76.
crossref pmid pmc
6. Kim YK. Recent trends and implications for digital healthcare [Internet]. Daejeon, South Korea: Institute of Information and Communications Technology Planning and Evaluation; 2018 [cited at 2020 July 11]. Available from: https://www.itfind.or.kr/WZIN/jugidong/1846/file986680593057551481-184602.pdf.

7. Kim JK, Lee SJ. Strategies and implications for ICT-based healthcare industry in major countries [Internet]. Sejong, South Korea: Korea Institute for International Economic Policy; 2016 [cited at 2020 July 11]. Available from: http://www.kiep.go.kr/sub/view.do?bbsId=global_econo&nttId=189741.

8. Scott RE, Mars M. Principles and framework for eHealth strategy development. J Med Internet Res 2013;15(7):e155.
crossref pmid pmc
9. World Health Organization. National eHealth Strategy Toolkit [Internet]. Geneva, Switzerland: World Health Organization; 2012 [cited at 2020 July 11]. Available from: http://apps.who.int/iris/bitstream/10665/75211/1/9789241548465_eng.pdf.

10. European Commission. 15th Meeting of the eHealth Network; Bucharest; 11–12 June 2019. [Internet]. Brussels, Belgium: European Commission; c2019 [cited at 2020 July 11]. Available from: https://ec.europa.eu/health/ehealth/events/ev_20190611_en.

11. Blumenthal D. Launching HITECH. N Engl J Med 2010;362(5):382-5.
crossref pmid
12. Neumeier M. Access 2022: setting new goals for digital health in Canada. Can J Nurs Inform 2019;14(1/2):1-2.

13. Kim SJ, An KS. An empirical study on efficient fiscal management for PPP (Public-Private Partnership) projects by using Delphi methods. J Public Policy Gov 2017;11(3):67-99.

14. Rowe G, Wright G. The Delphi technique as a forecasting tool: issues and analysis. Int J Forecast 1999;15(4):353-75.
crossref
15. Hsu CC, Sandford BA. The Delphi technique: making sense of consensus. Pract Assess Res Eval 2007;12(1):10.

16. Ludwig B. Predicting the future: Have you considered using the Delphi methodology. J Ext 1997;35(5):1-4.

17. Gordon TJ. The Delphi method. In: Glenn JC, Gordon, TJ, editors. Futures research methodology. New York (NY): American Council for the United Nations University; 1994. p. 1-30.

18. Delbecq AL, Van de Ven AH, Gustafson DH. Group techniques for program planning: a guide to nominal group and Delphi processes. Glenview (IL): Scott and Foresman; 1975.

19. Keeney S, Hasson F, McKenna HP. A critical review of the Delphi technique as a research methodology for nursing. Int J Nurs Stud 2001;38(2):195-200.
crossref pmid
20. Murry JW, Hammons JO. Delphi: a versatile methodology for conducting qualitative research. Rev High Educ 1995;18(4):423-36.
crossref
21. Diamond IR, Grant RC, Feldman BM, Pencharz PB, Ling SC, Moore AM, et al. Defining consensus: a systematic review recommends methodologic criteria for reporting of Delphi studies. J Clin Epidemiol 2014;67(4):401-9.
crossref pmid
22. Kim J, Kim H, Leem B, Yoon J. Analyzing the national medical service efficiency of OECD countries using DEA and malmquist productivity Index. J Korean Oper Res Manag Sci Soc 2012;37(4):125-38.
crossref
23. US Office of the National Coordinator for Health Information Technology (ONC). Federal Health IT Strategic Plan 2020–2025 [Internet]. Washington (DC): ONC; c2020. [cited at 2020 Jul 10]. Available from: https://www.healthit.gov/topic/2020-2025-federal-health-it-strategic-plan.

24. Hersh W, Williamson J. Educating 10,000 informaticians by 2010: the AMIA 10×10 program. Int J Med Inform 2007;76(5–6):377-82.
crossref pmid
25. Gibson CJ, Dixon BE, Abrams K. Convergent evolution of health information management and health informatics: a perspective on the future of information professionals in health care. Appl Clin Inform 2015;6(1):163-84.
crossref pmid pmc pdf
26. Luna D, Almerares A, Mayan JC 3rd, Gonzalez Bernaldo de Quiros F, Otero C. Health informatics in developing countries: going beyond pilot practices to sustainable implementations: a review of the current challenges. Healthc Inform Res 2014;20(1):3-10.
crossref pmid pmc
27. Marc D, Butler-Henderson K, Dua P, Lalani K, Fenton SH. Global workforce trends in health informatics & information management. Stud Health Technol Inform 2019;264:1273-77.
pmid
28. US Office of the National Coordinator for Health Information Technology (ONC). Certification of Health IT [Internet]. Washington (DC): ONC; c2020 [cited at 2020 Jul 10]. Available from: https://www.healthit.gov/topic/certification-ehrs/certification-health-it.

29. EuroRec. Promoting the use of high-quality electronic health record systems [Internet]. Ghent, Belgium: Euro-Rec; c2020 [cited at 2020 Jul 10]. Available from: https://www.eurorec.org.

30. Electronic Medical Record System Certification. The foundation of digital health care in Korea Electronic Medical Record System Certification [Internet]. Seoul, Korea: Korea Health Information Service; c2020 [cited at 2020 Jul 10]. Available from: http://emrcert.mohw.go.kr.



ABOUT
ARTICLE CATEGORY

Browse all articles >

BROWSE ARTICLES
FOR CONTRIBUTORS
Editorial Office
1618 Kyungheegung Achim Bldg 3, 34, Sajik-ro 8-gil, Jongno-gu, Seoul 03174, Korea
Tel: +82-2-733-7637, +82-2-734-7637    E-mail: hir@kosmi.org                

Copyright © 2020 by Korean Society of Medical Informatics. All rights reserved.

Developed in M2community

Close layer
prev next