PDF
ePub
Citation
Print
I. Introduction
Internationally, telemedicine has been the most controversial issue in healthcare fields. Although many countries, especially developed countries, tend to pursue telemedicine services aiming to improve healthcare accessibility and lower healthcare expenditures [1], telemedicine is not utilized and developed actively due to lack of legal and policy support for responsibility of services, absence of effective evidence, instability of information and communication technology, and insufficient infrastructure for education and training systems [23].
In particular, after the Korean government announced the revised medical law allowing physician-to-patient-based telemedicine, telemedicine has become a more divisive issue with physicians' firm opposition due to the insecurity of patients' health and other drawbacks. The Korean government supports telemedicine as an alternative to reduce the medical financial burden for sustainable management of the national healthcare industry according to changes of development in the national healthcare environment [4]. In this context, a sequential pilot project implementing telemedicine has been carried out in Korea [5]. Since then, as the Internet network and other information and communication technologies (ICTs) are rapidly growing, the government has begun to support physician-to-patient-based telemedicine. However, negative impacts, such as functional weakening of primary care and concentration of patients into large hospitals are concerns.
Physician-to-patient-based telemedicine is controversial among stakeholders because the prospects of telemedicine from all the parties are different. Moreover, the fundamental reason is that understanding and evidence of its advantages are insufficient. The government raises the aspect of advantages through implementing telemedicine, which are improvement of accessibility, cost saving, quality of medical service, and satisfaction of patients. On the other hand, the healthcare sector agitates against telemedicine if there are thought to be threats to the national health involved in the adoption of telemedicine. From the industrial perspective, the ICT industry is also contrary to healthcare sector, so academia focuses more on the problems generated by telemedicine. The most important point is that the public who are direct consumers of services have not had an opportunity to form or express their opinions regarding telemedicine. Inequality of accepting specialized information or knowledge about telemedicine, which includes complex and multifaceted characteristics, leads to the limitation of public understanding [56].
Therefore, there must be sufficient understanding of a comprehensive and multilateral evaluation framework to converge the opinions from the government, healthcare sector, industry, and academia, as well as the public. This paper suggests a comprehensive evaluation framework for telemedicine and telehealth system implementation and is intended to support related stakeholders' decision-making by promoting general understanding and resolving arguments and controversies. Specifically, the following research objectives were taken into consideration: 1) analyzing the existing alternative evaluation strategies, methodologies, frameworks, and approaches for telemedicine and telehealth in addition to those for health information systems in general, 2) gaining insight into multiple dimensions and factors that influence the quality and outcomes of telemedicine and telehealth systems by determining the attributes that may serve as barriers or facilitate the system, and 3) identifying common themes and issues in constructing an evaluation framework.
II. Scope of Evaluation Framework
1. Telemedicine and Telehealth
The terms telemedicine and telehealth tend to be used interchangeably, but there is general convergence of opinion that telemedicine is a subset of telehealth [7]. Both use "audio, video, and other telecommunications and electronic information processing technologies to provide health services or assist healthcare personnel at distant sites" [8]. However, while telemedicine refers to remote clinical services, such as consultation, diagnosis, treatment, and transfer of medical data, telehealth can refer to remote non-clinical services, such as administrative procedures, training, and education, in addition to clinical services [9].
However, more studies and published documents on telemedi cine are found than on telehealth because the clinical practice of telemedicine services involves more obstacles and barriers, such as licenses, reimbursement, etc. On the other hand, most success factors for telemedicine service can be expanded to telehealth since both systems use much of the same infrastructure of ICT equipment and networks, and they are used by the same types of professionals and consumers. Therefore, this study mainly emphasizes clinical telemedicine services for developing the factors of the evaluation framework.
2. Need for Evaluation Framework
We need a comprehensive evaluation framework for telemedicine and telehealth because the systems are very complex with multiple stakeholders and indirectly valued by multiple factors. Recently, Poultney [10] summarized the current problems for evaluating telemedicine and telehealth services with three issues: no universally accepted standard, varying weaknesses of current methodologies in terms of appropriateness, and limited coverage of evaluation focus on clinical-benefit and cost-benefit.
A variety of factors influence the implementation and operation of telemedicine services. In general, the system requires administrative, organizational, behavioral, technological, sociological, financial, and political justifications. Therefore, telemedicine services should be assessed continuously and multi-dimensionally from the planning stage to performance evaluation. In addition, a number of sub-factors for each dimension of the service should be considered for the evaluation framework. For example, for the economic aspects, measurements should include evaluations of not only costs and benefits for multiple stakeholders but also insurance coverage and reimbursement.
3. Approaches to Comprehensive Evaluation
Telemedicine services are complex in nature compared with traditional provider-to-patient on-site healthcare delivery services. Therefore, evaluation frameworks for telemedicine can also be complex with many potential inputs, activities, outputs, outcomes, and stakeholders. This may be one reason for a lack of established telehealth evaluation protocols [11]. There is no standardization of definitions, criteria, and measures across the literature, which leads to ambiguity and confusion. Moreover, the methodologies for conducting telemedicine evaluations are not consistent. In addition, since there is no link between the telemedicine evaluation criteria and health performance indicators, the overall impact of telehealth on the healthcare system cannot be judged [12].
There have been a variety of researches conducted on the evaluation of telemedicine. However, some have focused on particular clinical specialties while others focus on specific outcomes such as cost-effectiveness, patient satisfaction or clinical functional outcomes, etc., and a few have concentrated on frameworks and guidelines to evaluate and support telemedicine [1213]. This study focused on a comprehensive evaluation framework by summarizing themes across the range of evaluation techniques. The framework will make it more efficient to undertake evaluation of any telemedicine implementation, to produce more widely applicable findings, to share these and to improve practice based on the collective results. A conceptual framework will be provided to incorporate the key dimensions, criteria and measures that need to be considered in the evaluation of telehealth implementations for a formal structured approach to the evaluation of a service.
III. Review of Existing Evaluation Approaches
Ammenwerth and de Keizer [14] suggested that an evaluation of information systems in healthcare fields must consider multiple stakeholders with heterogeneous perspectives: the actors such as people, organizations, and community; and artifacts such as technology, the environment (including social and political characteristics), and their interactions. Evaluation should also consider outputs and outcomes for immediate results and intermediate goals, respectively. Therefore, evaluating telemedicine entails incorporating all these aspects with medical, technical, psycho-social, organizational, business, political, and societal factors. In response, this study attempted to develop a comprehensive framework by extracting and combining elements from existing foundational evaluation approaches.
1. Foundational Framework for Information Systems in General
Basically, it is important to consider various quality improvement techniques to check the effectiveness, efficiency, and safety of the system and its processes. While there are many techniques and tools for quality improvement in healthcare, the fundamental framework for the quality of healthcare is found in Donabedian [15]. The model observes the structure, processes, and outcomes of a service. Structure measures cover the accessibility, availability, and quality of resources; process measures include the delivery of health care services by clinicians and providers; and outcome measures assess the final result of healthcare, which can be influenced by environmental and behavioral factors. In the context of information systems, the framework of structure, process, and outcomes is presented in Table 1 as an example. DeLone and McLean [16] considers structure measures with the quality of information and system, process measures with system use and user satisfaction, and outcome measures with individual and organizational impact.
From the management information system (MIS) perspectives, the conceptual model for MIS evaluation assesses attitudes, value perceptions, information usage, decision performance on the process from need assessment, through MIS design and development, to system implementation [17]. Considering the popular technology acceptance model (TAM) [18], the model has a strong user evaluation approach to acceptance of a technology or service, but no clinical, organizational acknowledgement; hence, it is criticized for perceived ease-of-use being a relatively poor indicator for acceptance. For the security assurance outcome in particular, Chaula et al. [13] proposed an information security assurance framework considering non-technical factors as well as technical factors along the assurance life cycle. The proposed framework covers policy, design, implementation, and operation assurance for a non-trusted system to be changed to a trusted system.
2. Framework for Information Systems in Healthcare Fields
The unified theory of acceptance and use of technology (UTAUT) model was developed to understand the complexity of determining healthcare professionals' acceptance and use of new ICT [19]. The model considers four core constructs, which impact a user's behavioral intention, including performance expectancy, effort expectancy, and social influence, as well as facilitating conditions. Another generic ICT evaluation framework is found in the clinical, human and organizational, educational, administrative, technical, social (CHEATS) approach [20]. CHEATS covers a broad spectrum of multi-dimensional perspectives, especially human and organizational factors. More practically, the factors related to electronic medical record technology acceptance by healthcare professionals include the aspects of technological system that are deployed, organizational compatibility, and user technology skills, as shown in Table 2 [21].
3. Evaluation Framework for Telemedicine
More specific evaluation frameworks have been proposed for telemedicne and telehealth systems. Ohinmaa and Reponen [22] reported a five-dimensional assessment model including performance measures (time, quality, cost); outcome measures (safety, efficacy, effectiveness); summary measures (cost comparisons); operational considerations (access, acceptability); and other issues (confidentiality, legal). Bashshur et al. [23] defined a model using three dimensions based on functionality, technology, and applications; explicitly, functionality (consultation, diagnosis, monitoring, mentoring), technology (modes, network design, connectivity), and applications (treatment modalities, medical specialty, disease types, sites).
The Institute for a Broadband-Enabled Society (IBES) also reported a unified approach for the evaluation of telehealth implementations in Australia, using patient control, clinician quality of care, organization sustainability, and technology capability [24], with a strong approach to finance related aspects and without consideration of human factors. On the other hand, the telemedicine maturity model (TMMM) was proposed with emphasis on the process with multiple maturity levels of a telehealth service [25].
Rather recently, Nepal et al. [26] presented a broader and integrated model with six major components for delivery design, implementation, and evaluation of telehealth services. The six key dimensions are health domains, health services, technologies, communication infrastructure, environmental settings, and socioeconomic analysis. Example components for each dimension are summarized in Table 3.
IV. Summary of Evaluation Framework for Telemedicine
1. Outcome of Telemedicine
An evaluation framework for telemedicine can be considered to be the examination of the quality and safety outcomes and the influence of telemedicine services on these aims for healthcare. If any telemedicine service has objectives concerned with quality, the main frameworks for quality use the key dimensions of care such as effectiveness of service, appropriateness of means of delivery, acceptability, efficiency, and equity. On the other hand, if any telemedicine service has objectives concerned with safety, the main frameworks use the dimensions of impact of medical error, type of process failure, setting of incident occurrence, and cause or factors leading to safety incidents [27].
For both outcomes, the following common dimensions are considered for the comprehensive evaluation framework: human participant-related aspects, healthcare service process, organizational perspectives, administrative work, technological aspects, ethics and policies, etc. For example, human participants comprise patients, clinicians (including doctors and nurses), support personnel (including technicians or laboratory staff), and management personnel (including administration, technology providers, etc.).
2. Dimensions of Successful Telemedicine Implementation based on Barriers
Many published studies have shown positive healthcare outcomes of telemedicine through enhanced communication with health providers and closer monitoring [28], but telehealth also creates some new risks and challenges along with new opportunities for the healthcare industry. Therefore, telemedicine must still overcome a variety of obstacles and barriers before it reaches its full potential. A few studies have addressed licensure and reimbursement issues along with the development of telemedicine care standards as major obstacles [29]. However, more barriers to the implementation of telemedicine need to be recognized in a comprehensive evaluation framework with a broad spectrum of issues and obstacles. The barriers can be classified into the following five categories: behavioral, organizational, technical, economical, and legal [30]. Representative examples for each category are introduced as follows.
1) Behavioral perspective
If end users, patients and healthcare providers, of the services are not comfortable with the technology, it is difficult to use or it does not fit into their workflow, adoption may be expected to be difficult. This behavioral category includes factors such as users' empowerment, acceptance, and fear of being replaced by technology. It has been observed that "the success of a new telehealth service depends far more on changing the clinicians and patient's expectations and patterns of behavior than on technology" [31]. Lack of training, education, and technical assistance to telemedicine users has been noted as a major obstacle to the effective use of advanced technology equipment and networks [32].
2) Organizational perspective
It is crucial to integrate telemedicine services into existing organizational structures and to provide institutional support to implement these services [33]. Whitten and Allen [34] identified significant organizational communication deficits, including an "absence of explicit strategic goals, perceived central leadership, lack of information about technology, poor design for scheduling and utilizing technology, and poor communication with medical personnel and the public, etc." Innovations in organizational communication and structure should follow innovations in telemedicine technology.
3) Technical/knowledge perspective
To be successful, telemedicine needs systems with technology and knowledge to "consistently and reliably review patient data and alerts and the development of appropriate algorithms to respond to patient data in a manner that improves patient outcomes" [30]. Ineffective and inefficient operation of the resources that are available often results from limited availability of an information network or a lack of coordination in infrastructure. In addition, lack of evidence to support the belief that telemedicine provides cost-effective services is also a significant barrier to the expansion of telemedicine [8].
4) Economic perspective
Reimbursement policies constitute one of the most important issues for telemedicine implementation. In particular, insurance, especially private insurance, tends to be reluctant to pay for telemedicine services. In the long run, the success of telemedicine is almost impossible without adequate third-party reimbursement. On the other hand, the perspectives of physicians and insurers are also a significant issue to be considered because they are at opposing positions in terms of costs and benefits of telemedicine. Therefore, economic perspectives should be considered at individual, organizational, and societal levels, respectively.
5) Rules and regulations perspective
Medical liability and malpractice issues are the significant before hospitals adopt telemedicine. In particular, if hospitals use telemedicine to treat patients in multiple geographical areas, clinicians have to make sure they are complying with multiple sets of laws [35]. Issues regarding the security of medical information can also be a significant barrier to telemedicine implementation [35]. In addition, since telemedicine services hold and transmit rather sensitive patient data electronically, hospitals need to keep electronic health information secure and ensure that only the right people have access to the data. Although medical records from traditional healthcare delivery have ongoing privacy and confidentiality issues, the adoption of ICT incurs additional risks in keeping information secure. On the other hand, interoperability issues among ICT equipment and networks are a potential barrier due to a lack of standards.
3. Comprehensive Evaluation Framework
Considering existing evaluation frameworks for information systems in general, health information systems, and telemedicine and telehealth only, the proposed dimensions and their components are summarized and integrated with five dimensions of barriers. In particular, while a longitudinal framework with macro perspectives is presented with a logical framework approach, a horizontal and cross-sectional framework with micro perspectives is presented with a fishbone diagram.
1) Logical framework approach for telemedicine
While the logical framework approach (LFA) is an analytical process and a set of tools used to support project planning and management with a long tradition in many other fields, it is not commonly applied to the system development and implementation for health information systems. According to the World Bank [36], "the Logical Framework has the power to communicate the essential elements of a complex project clearly and succinctly throughout the project cycle. It is used to develop the overall design of a project, to improve the project implementation monitoring and to strengthen periodic project evaluation". In this context, the LFA is required for telemedicine and telehealth services since very complex and difficult issues with multiple stakeholders are involved. However, the formal procedure for the LFA requires stepwise application of a stakeholder analysis, problem analysis, objectives analysis, alternative analysis, etc., to derive the final summary matrix. Therefore, this study shows and summarizes only evaluation steps of the project cycle in Figure 1, as the basis for preparation of the LFA. The general steps consist of inputs, activities, outputs, outcomes, and impact in longitudinal order, and representative objectives and components in each step are presented.
2) Fish-bone diagram for telemedicine
As a visualization tool for identifying the root causes of quality problems, the fishbone diagram summarizes potential causes for an effect or problem by sorting possible causes into categories [37]. The factors and barriers of telemedicine implementation described in the previous section were reorganized and sorted with the main skeleton and subsequent bones of the diagram in Figure 2. The main skeleton consists of three broad column categories with six dimensions: 1) human (service provider and patients/clients), 2) system (organization and technology), and 3) environment (society and rules/regulation). For each of six dimensions, the potential causes of significant factors and barriers that can influence the outcomes are summarized in the diagram. Of course, the branches and sub-components presented in the diagram are exemplary, and not limited for the comprehensive evaluation framework.
V. Conclusion
Telemedicine and telehealth are explosive technological areas that appear to threaten traditional healthcare delivery services while offering the potential to reform the healthcare industry by reducing costs and increasing quality and patient satisfaction. Therefore, they are rapidly progressing toward an essential core of healthcare delivery. However, there is still controversy about the performance evaluation of telemedicine, for which enormous amounts of money and information technology have been invested. Stakeholders who participate in telemedicine have various conflicting opinions and ideas about outcomes and characteristics of services. To bring consensus among their opinions, an evaluation framework is required to verify the effectiveness and stability of telemedicine to provide services taking into account behavioral, administrative, technological, social, and policy perspectives. However, it is not easy to develop a monitoring and evaluation framework for any project concerned with innovative services such as telemedicine and telehealth. This study organized a basic evaluation framework generally applicable through articles and diverse cases of telemedicine and telehealth. Evaluation factors concerning the aspects of information technology, the satisfaction of service providers and consumers, cost, quality, and information security were organized using the fishbone diagram. To implement telemedicine services, stakeholders should make decisions based on sufficient evidence of effectiveness and stability measured by the comprehensive evaluation framework. Further work would be valuable in applying more comprehensive evaluations to verify and improve the comprehensive framework across a variety of context with more factors and participant group dimensions.
XML Download