Real-world data (RWD) refer to data collected from diversified areas of daily life that are outside the scope of highly controlled randomized control trials (RCTs).1 Examples of RWD are the large quantities of data collected broadly through the use of a diverse range of medical devices by various users, including self-checked glucometer or sphygmomanometer. These real, practical data cannot be obtained through contemporary or traditional clinical trials. A wide range of data already accumulated, including the electronic medical records (EMRs) used in hospitals,234 Korea National Health and Nutrition Examination Survey (KNHANES) data,5 Health Insurance Review and Assessment service data6 or Korea Adverse Event Reporting System (KAERS) database7 are also included in RWD.

EMR, in allowing for the accumulation of a large amount of clinical data, are increasingly used for medical research.89 Worldwide, EMR-based clinical research has been reported to increase the quality of research and satisfaction of clinicians.10111213 Depending on study design and objectives, EMR research may utilize a variety of data: medication prescription, treatment decision making, disease management, and clinical research data. For these reasons, EMR data are recognized as definitive data with the highest reliability among RWD. Analysis of RWD yields real-world evidence (RWE), and many scholars are focusing their interest on RWE research using RWD accumulated broadly, along with the issue of big data. Efforts have been made toward expanding the range of relevant clinical research.

Fundamentally, it is desirable for the new research format, RWE, and the traditional research format, RCT, to be mutually complementary. However, despite the comprehensive reports on the importance of RWE, there is a tendency among scholars of mistrusting the findings of RWE compared with RCT. For many researchers, overcoming a diverse range of biases that can occur in EMR-based RWE research requires trial and error. Therefore, we intend to present opinions on precautions that can be used effectively in executing research using RWE from the clinical rather than technical perspectives.

RCT conducts research on highly selective populations and is managed in tightly controlled settings.14 RCTs are conducted under the premise that the research outcomes of the chosen sample population represent those of the entire population. This research format is deemed to have the highest reliability. The majority of clinical guidelines are given on the basis of the resultant values deduced from RCTs.15 However, a disadvantage is that participants may fail to reflect the actual clinical site sufficiently to represent the entire population, given the clear set of inclusion or exclusion criteria. This drawback of exclusion has been deemed a bias.1617 That is, there would be difficulty in applying the resultant values of data analysis confirmed in a small number of patients selected in accordance with set criteria in RCTs to diversified actual situations that could occur in the real world. For this reason, the foundation on which to elevate definitive reliability through concurrent execution of RWE research, in addition to RCT, needs to be established; these two should ideally be developed in a mutually supplementary rather than competing relationship. For example, if the RCT concentrates on the efficacy of a drug, the RWE can focus on the epidemiology, effectiveness, safety, or costs of treatment related to the drug (Tables 1 and 2).18 RWE research that can sustain follow-up observation after the completed RCT research holds important significance. Clinical trials and secondary syntheses of evidence-based information can be biased owing to undisclosed financial and nonfinancial conflicts of interests. Therefore, research should be undertaken with the perception that managing these conflicts of interests more thoroughly is crucial to maintaining the essence and credibility of research.19

Previous research on the use of EMR data have focused on their convenience and accessibility.20 With the recent accumulation of extensive data in EMR systems, these data can be used in clinical studies. The use of EMR-based RWE addresses the disadvantages of both cohort studies (e.g., follow-up cost, long study periods, and maintenance of consistency during the study period) and RCTs (e.g., loss of participants at follow up, changes in treatment, long study periods, and expense). In EMR-based clinical research, the investigator has direct access to the EMR system and can quickly verify a hypothesis by sampling related variables from the system.

Accumulation of large-scale patient data and clinical trials using such data raise the important issue of the protection of personal information. In the case of Korea, numerous limitations on the use of clinical big data have been enacted to address issues related to personal information leaks. Therefore, privacy protection must be mentioned clearly when preparing works on RWD. This practice must include the fact that the name and resident registration number will not be included in the accumulated data, and information directly related to privacy, such as address, telephone number, and e-mail, will be removed. Any combination that can lead to the identification of an individual, such as hospital registration number and date of birth, must be removed immediately prior to the stage of analysis. Further, all files on the extracted information must be encrypted and then stored in an encrypted computer, accessible only to the designated researcher. Additionally, contents of theses must not deal with the personal information of patients; researchers should emphasize that the research has no possibility of causing additional physical harm whatsoever to the subjects. Finally, it must be specified that the advance consent of the patient is not necessary given the anonymity of the subjects through encryption of data and that there is no effect on the rights and welfare of the subjects because past data, collected after having completed treatment, are used (informed consent not required). The statement “the research was conducted after having acquired the approval of IRB” must also be included.

Compared with RCT, EMR-based RWE study poses difficulties in controlling for biases that may affect the study outcomes. Research projects need to be commenced with a definitive design to minimize such biases. Data quality management (DQM) is the most important part of the extracted data in RWE. As extracted data could be deemed byproducts of the process of treating patients, rather than data elaborately organized and structured in accordance with the purposes of the research, the quality of the data can be low. There may even be many errors or erroneous inputting in the course of their collection. Through DQM, comparison with original data can be confirmed, including the mixed use of numbers and letters, erroneous descriptions, and serious errors. This phase may be time consuming, depending on the number or capacity of data. The most serious bias normally occurs at this stage.

As mentioned above, it is important to secure larger samples size to elevate fundamentally the reliability of RWE. Although larger quantities of data do not necessarily elevate accuracy and reliability, using the same would help minimize bias that occurs in the smallest number. Even with plenty of actually extracted data, a researcher may end up excluding a large number once the data are sorted in accordance with the purpose of a research. For example, there are many cases in which the basic baseline results, such as height/weight and blood pressure, end up omitted. Without baseline values, there can be no follow up, and changes in clinical laboratory data cannot be examined. There also are frequent cases in which the actual overall number of research subjects is reduced by more than half, such as when patients for whom follow up is not possible or patients who transferred to another hospital in the middle of the research are excluded. The exclusion of a large number of samples is in itself a significant bias, and it is the aspect of EMR-based clinical research pointed out most frequently by journal reviewers as a hard limitation.

Certainly, not all the data are available in EMR. For one, it is difficult to confirm the compliance of the subjects using EMR, including whether they took drugs as instructed in a drug-related research. Hence, assessment of the effectiveness or efficacy of drugs tends to be difficult in RWE research (This is generally carried out in RCTs). Checking whether subjects took the drugs as instructed by checking the number of pills prescribed and those remaining at the time of the next visit can be implemented as an alternative method for assessment of the compliance of subjects. Failure of the medical staff to make entries in the medical chart is also a problem. Proper research would be possible if the medical staff describe the state of patients accurately and in detail. Further, although patient conditions and treatments can be described, EMR cannot be used to elucidate scientifically causal relationships.

The domain of RWE in addition to EMR data mentioned in this work is extremely broad. It can be collected directly from the patient by means of a diverse range of medical devices and by utilizing a wide range of data. Accordingly, RWD are increasing in volume, whereas RWE is being gradually expanded, with continuous advancement of their respective analytic functions and protocols. In addition to the importance of the quantity of data, it is necessary to concentrate on the means of properly structuring and utilizing rapidly accumulated clinical data. RWE and RCT research are not in a mutually competitive relationship. Presently, RWE research cannot substitute RCT research. However, RCT research, bestowed the highest reliability, and RWE research, perceived to reflect the actual clinical aspect, can be proven to share a mutually supplementary relationship to become the most powerful resource in evidence-based medicine.