Pragmatic Clinical Trials for Real-World Evidence: Concept and Implementation

Na-Young Jeong; Seon-Ha Kim; Eunsun Lim; Nam-Kyong Choi

doi:10.36011/cpp.2020.2.e12

Journal List > Cardiovasc Prev Pharmacother > v.2(3) > 1516079356

Go to TopGo to Top Go to BottomGo to Bottom

TOOLS

Jeong, Kim, Lim, and Choi: Pragmatic Clinical Trials for Real-World Evidence: Concept and Implementation

Special Article

Cardiovasc Prev Pharmacother 2020;2(3):85-98.

Published online: 31 July 2020

DOI: https://doi.org/10.36011/cpp.2020.2.e12

Pragmatic Clinical Trials for Real-World Evidence: Concept and Implementation

Na-Young Jeong, BS

, Seon-Ha Kim, BS

, Eunsun Lim, BS

, Nam-Kyong Choi, PhD

Department of Health Convergence, Ewha Womans University, Seoul, Korea

Correspondence to Nam-Kyong Choi, PhD Department of Health Convergence, Ewha Womans University, 52 Ewhayeodae-gil, Seodaemun-gu, Seoul 03760, Korea. E-mail: nchoi@ewha.ac.kr

Received 30 June 2020 Accepted 28 July 2020

This is an open-access article distributed under the terms of the Creative Commons Attribution Non-Commercial License (http://creativecommons.org/licenses/by-nc/4.0/) which permits unrestricted non-commercial use, distribution, and reproduction in any medium, provided the original work is properly cited.

Abstract

The importance of real-world evidence (RWE) has been highlighted in recent years, and the limitations of the classical randomized controlled trials, also known as explanatory clinical trials (ECTs), have been emphasized. Post-marketing observational studies have several problems, such as biases and incomparability between patient groups, and RWE can only be obtained after a certain period. Therefore, pragmatic clinical trials (PCTs) have garnered attention as an alternative to obtaining scientifically robust RWE in a relatively short time. PCTs are clinical trials that have a pragmatic concept, i.e., the opposite of ECTs and are intended to help decision makers by evaluating the effectiveness of interventions in routine clinical practice. The characteristics of PCTs are the inclusion of various patients in clinical practice, recruitment of patients in heterogeneous settings, and comparison with actual clinical treatments rather than a placebo. Thus, the results of PCTs are likely to be generalized and can have a direct impact on clinical and policy decision-making. This study aimed to describe the characteristics and definitions of PCTs compared with those of ECTs and to highlight the important considerations in the planning process of PCTs. To perform PCTs for the purpose of obtaining RWE, the contents covered in this study will be helpful.

Keywords: Causality, Epidemiologic studies, Patient outcome assessment, Pragmatic clinical trials

INTRODUCTION

Traditionally, randomized controlled trials (RCTs) in controlled settings have been considered the gold standard for studying the efficacy of health interventions.1) It is the backbone of clinical research because randomization in well-defined and controlled settings can improve the internal validity and reduce bias, but these settings tend to overestimate the benefits and underestimate the risks.2) Recently, there has been great interest in conducting post-launch observational studies that increase the external validity using real-world data (e.g., claims data and electronic health records [EHRs]) including a large population.1)3) However, several challenges have been reported when conducting these studies, including bias (e.g., selection bias), inaccuracy of diagnostic codes, and incomparability between patient groups.4)5) In terms of real-world evidence (RWE) delivery, pragmatic clinical trials (PCTs) are proposed as a method to provide evidence in a relatively shorter time to solve problems faced by multiple stakeholders in the real world.6) PCTs combine the scientific rigor of RCTs with the real-life application of observational studies to suggest better solutions that reflect daily clinical practices.6) PCTs randomize patients and evaluate the effectiveness rather than efficacy of the intervention (treatment) to investigate the validity of the intervention in a broad routine clinical setting.7) PCTs are also referred to as pragmatic randomized clinical trials or embedded pragmatic clinical trials (ePCTs). With the recent increase in the importance and availability of RWE, interest in PCTs has increased worldwide. When searching using PubMed, more than 600 of the included trials conducted between 1977 and 2017 were “pragmatic” in their titles, while more than half of them were concentrated from 2014 to 2017.8) In Korea, the National Evidence-based healthcare Collaborating Agency organized a research center to support and fund trials including PCTs in 2019, which has identified the effectiveness of health care technologies and provided evidence for regulatory decisions. Given these circumstances, it is necessary to understand the concept and implementation of PCTs. In this study, we briefly described the main ideas and planning stages of PCTs and the key considerations required to perform them.

CONCEPT OF PCTs

The concept of pragmatism in trials was first presented by Schwartz and Lellouch in 1967.9) They classified randomized trials into 2 categories, “explanatory” and “pragmatic,” depending on the conditions of comparison and the level of current practice. Explanatory clinical trials (ECTs) are intended to provide information to understand the efficacy and biological mechanisms of interventions by comparing them in the identical and strict conditions under controlled settings.9) By contrast, PCTs aim to assess the effectiveness of interventions in the usual clinical setting and to make relevant decisions.7)9)

After the initial concept of PCTs was presented, a variety of factors have been presented by several authors to distinguish ECTs from PCTs (Table 1).10-16) ECTs determine whether the intervention actually work in controlled environments, while PCTs assess whether the intervention work in real-world settings.14) Other differences between the 2 representative trials are as follows:

Participants: ECTs use high levels of inclusion and exclusion criteria to select the appropriate research targets. However, PCTs minimize the exclusion criteria to include a variety of patients visiting the routine clinical sites.
Settings: ECTs are performed in strictly controlled settings to minimize bias. By contrast, PCTs are performed in multiple institutions that cover the full range of actual clinical trials, not in a single center, to maximize applicability and generalizability.
Intervention: ECTs provide specific directions, so that the delivery of intervention and compliance with intervention are closely monitored. However, PCTs leave the providers with details of the implementation of intervention as in the normal practice and do not require compliance with the interventions.
Comparator: ECTs recommend that the therapeutic impact of experimental interventions be compared with placebo, but PCTs generally compare the relative effectiveness of the treatment studied with the treatment currently used in practice instead of the comparing it with a placebo.
Outcomes: PCTs focus on a broad set of patient-centered outcomes or endpoints; however, ECTs have limited or surrogate measurable outcomes, especially clinical symptoms or biological markers. Consequently, PCTs have high relevance to real life and direct social value than ECTs because they are more likely to be applied to practice at once.17) Based on these factors, Califf and Sugarman11) defined PCTs as “designed for the primary purpose of informing decision makers regarding the comparative balance of benefits, burdens, and risks of a biomedical or behavioral health intervention at the individual or population level”.

Explanatory and pragmatic trials should not be considered as completely dichotomous, but should be seen as two extremes within the concept of a continuum (Figure 1).18) To help design the trials that match the researcher's initial intentions by clarifying the concept of explanatory and pragmatic, the PRagmatic Explanatory Continuum Indicator Summary (PRECIS) tool was developed in 2009.19) In 2015, a revised version, PRECIS-2, was introduced with more loosened requirements.18) Investigators can identify the nature of the clinical trials they perform by checking for explanatory/pragmatic continuum in nine domains (Table 2).20)

STEPS IN CONDUCTING PCTs

The National Institutes of Health (NIH) Collaboration in the US produced the “Living Textbook” to guide the overall considerations when performing PCTs. In particular, the NIH provided essential steps in conducting PCTs using the ePCT Quick Start Guide for convenience. The next 9 steps in the guide represent what the researchers need to consider from planning to initiate the PCTs.

Learning what makes PCTs different

The first step is that researchers understand the characteristics of PCTs. The researchers are required to have abilities to understand the differences between PCTs and traditional ECTs, and to identify the pragmatic components in PCTs by considering the balance between flexibility, adherence, and generalizability. The typical research question in PCTs is whether specific treatments in routine clinical practice are effective in improving the health outcomes in patients compared with other general treatments.21) Research questions are likely to be applied to practices immediately16) because PCTs have direct social value compared with other types of trials, as the questions originate from a variety of stakeholders such as patients, medical personnel, and delivery systems.21)

Building partnership to ensure a successful trial

In order to conduct a research that supports the objectives of each stakeholder, the participation of key stakeholders in the entire process of PCTs is essential.22) PCTs generally use EHR data from multiple institutions, which makes it difficult to achieve without partnerships.23) The main potential stakeholders of patient-centered research such as PCTs include patients, clinicians, health care organization leaders, researchers, policy makers, and regulators.22) For the partnership before carrying out PCTs, researchers and health care providers should understand and respect the demands, goals, and tasks of each other.23) Stakeholders participate in PCTs in a number of ways, including advisory groups, collaborations, and co-design.24)

Planning for sustainability before launching of PCTs

The investigators need to develop a sustainable plan before the launch of PCTs. Before the designing of trials, researchers have to understand the process and key considerations related to the performance, dissemination, and continuance of the trials. These considerations include the impact of findings, requests of stakeholders, the best trial designs, and the resources needed to deliver the intervention.25) Researchers should develop overall plans for data sharing collected through PCTs and consider the precautions of sharing patients' data to protect their personal information.26)

Choosing the right outcomes

The biggest feature of PCTs is that the outcome being selected should be highly relevant to patients, clinical physicians, communities, and real-world clinical decisions. Therefore, the outcome should be discussed with relevant stakeholders in advance.9)27) The inappropriate outcomes in PCTs, for example, include surrogate outcomes, physiological outcomes, and unimportant outcomes for patients because they have unclear relationships with the endpoints in routine clinical practice and are not suitable to support the actual clinical decisionmaking.28)29) After selecting the results and endpoints to be assessed in PCTs, the researchers would investigate routinely collected outcomes from a variety of resources, including EHRs and electronic case report forms while minimizing the interference of the clinical practices.27) However, since information in EHRs may be inaccurate or missing, the validity of the algorithm of the definition of results must be verified and the patient-reported outcomes (PROs) directly provided by the patients must be used.12) PROs are important source of information to understand how treatment improves a patients “quality of life and the results that represent a patients” subjective experiences.30) The selection and measurement of outcome should be accurate and reliable without interference in the clinical practices.27)31)

Planning the study design and analysis

Researchers should plan the overall study design such as study settings, randomization, and blinding methods suitable for PCTs and perform a statistical analysis following this step.

Study sites: The sites in which PCTs are performed should be similar to the actual clinical setting and should be representative.12) Sites should be carefully selected because the prevalence of diseases and the availability of medical care may vary greatly depending on the type and region of the institutions.32) In order to select the sites for PCTs, the heterogeneity, patient characteristics, and disease prevalence of the sites should be assessed.32) Then, the feasibility and capacity of the study should be confirmed, including interest in study objectives, level of study experience, and number of patients accessible.32) PCTs should include several heterogeneous sites because if only single or homogeneous sites are included, the test results may not be applicable to patients in various settings.
Randomization: Randomization can help reduce the selection bias caused by the differences in study groups; any difference in the outcomes can be attributed to the assignment of treatments. Therefore, randomization should be applied in RCTs including PCTs.33) Even though randomization at both the patient level and cluster (e.g., sites and physicians) level12) is possible in PCTs, randomizing at the patient level may affect patients to select intervention, which may result in low recruitment rates.34) Therefore, cluster randomization, which involves the assignment of treatments at the level of physicians, hospitals, or other units handling multiple patients, is useful.34) There are several types of cluster randomization: cluster with crossover randomized design, cluster with partial crossover randomized design, and simple cluster-randomized design.35)
Blinding: Blinding of patients and health care providers during the treatment period is useful to minimize bias and enhance the internal validity in most clinical trials such as ECTs. In PCTs, it is generally not appropriate to use placebo and blinding owing to ethical reasons, preference of non-blinded trials by patients, and the specificity of PCTs.16) Therefore, the following methods can be recommended to minimize bias when patients or investigators are not blinded in PCTs12): 1) use of hard endpoints (i.e., objective endpoints) rather than subjective endpoints, 2) determination of endpoints from blinded health care professionals, and 3) blinding of statisticians and data analysts during the trial and planning of analysis, if possible. If blinding is not possible for the individual patients and investigators, the cluster randomization method can be used.36)37) The lack of blinding in PCTs decreases the internal validity, but it will be easier to generalize the test results by increasing the external validity.16)
Statistical analyses: The statistical analysis methods used mainly in PCTs must be fully understood, and an analysis plan should be made. The analysis method of RCTs can be applied to PCTs, but considerations are required because of the uniqueness of PCTs.38) Intention to treat (ITT) analysis and per-protocol (PP) analysis are required in analyzing the results of clinical trials, and ITT is considered the principal analysis in PCTs.39) Although problems of validity may occur if the randomized patients switch to other alternative treatments during the ITT analysis, ITT is generally recommended as it reduces the risk of selection bias and the unguaranteed covariate balance that can occur in PP analysis.38) However, large amounts of missing data in the ITT analysis can cause a confounding bias.38) To solve this, a follow-up history and the possible reasons for missing data should be recorded for all patients, which is the basis for analyzing the extent and the pattern of missing data. Bias because of missing data can be supplemented using causal inference tools such as a marginal structural model or through inclusion of additional data such as PROs from other sources (e.g., wearable devices).12) In addition, PCTs include more heterogeneous populations than RCTs; hence, the responses of subgroups to interventions may differ.38) When evaluating the effectiveness of treatments in subgroup analyses, a priori subgroup analysis with a large sample size should be used to ensure valid estimates of treatment effects and avoid obtaining false positive results.

Considering oversight and monitoring

In this phase, it is necessary to consider whether provision of an informed consent and a data monitoring committee (DMC) are needed to protect the participants of PCTs and to monitor the entire trial. Informed consent is usually required in clinical trials to respect and protect the study participants' privacy. However, an in-depth research consent that is commonly used in traditional RCTs can cause a selection bias and obstruction of real-world clinical practice.40)41) Thus, several alternatives were proposed to better integrate consent procedures within a routine clinical setting in PCTs. First, “integrated consent” proposes an approach that integrates clinical and research consent within the same clinical encounter.42) Physicians record the theoretical basis of treatment, the description of the potential risks and benefits of the treatment, and the results in the EHRs and provide normal treatment based on clinical practice. “Targeted consent” involves a process in which the study patients provide minimal information for consent during the consent procedure, owing to which patients can decide whether to register.43) The broadcast approach uses notifications located in prominent locations to inform patients regularly that RCTs may be performed occasionally; therefore, explicit consent is not mandatory.44)45) Finally, a “waiver of consent” involves a process in which the participants are not informed that a study will be conducted and they do not decide whether to participate in the study,41) which is allowed only in highly limited situations and is generally not performed in PCTs.

In the case of PCTs, the risk-benefit balance and data integrity changes are not only monitored but also special considerations may be applied owing to the practical characteristics of a routine clinical care. DMC is a committee that operates independently of researchers and sponsors in clinical trials.46) Investigators should decide whether to use DMC to monitor trials while protecting the study participants.47) The US Food and Drug Administration guidelines48) recommend the use of DMC if clinical trials have any of the following characteristics: long-term and large-scale multicenter studies, specific safety issues known in advance, and the inclusion of potentially vulnerable populations. The use of DMC in PCTs will ensure scientific validity, as PCTs are mostly multicentered studies comprising a large number of patients from several institutions.47)

Testing the feasibility of trials

This step is to assess the feasibility of PCTs, which serves to link the actual test design to their performance in the real world. The 2 elements of evaluating feasibility are as follows: evaluating the logistics of clinical trials within the existing health care system and conducting pilot tests on the key aspects of PCTs.49) The logistics assessment of PCTs involves documentation of a comprehensive record of PCTs and investigation of the training required for stakeholders.50) Pilot testing includes consideration of methods such as access to actual EHR data usage, cooperation with statistical experts to predict problems related to cluster randomization, and consideration of the valid definition of outcomes of interest.51) The effects of factors related to PCTs, such as informed consent forms, should be assessed.51) This process confirms the feasibility by exploring potential problems and solutions when PCTs are actually performed.

Preparations for the launching of PCTs

After assessing the feasibility of PCTs, it is necessary to adjust the research design, interventions, and human resources. For example, problems that PRO data are not being routinely collected in the EHR system and problems with the availability of personnel to provide the intervention to patients can be found, which can cause changes in the research flow.52) Hence, a readiness checklist must be implemented to determine whether PCTs can be launched.53)

Recruitment plans for the enrollment of participants in PCTs should be devised in this phase. PCTs are recommended to have a well-representative cohort and should include patients representing the real world to enhance the generalizability of the results.16) For this, researchers should set wide inclusion criteria and minimum exclusion criteria; thus, patients are not excluded from the study even if they have comorbidities or receive concomitant drugs. Unlike other traditional clinical trials, some degree of patient co-enrollment is acceptable in PCTs owing to the improvement in enrollment even though this can lead to unintended interaction and affect the power of the trials.34)54) In addition, vulnerable populations such as children, pregnant women, and those with disabilities should not be excluded from PCTs because excluding them makes it difficult to identify the actual benefits and risks of the treatments in these patient groups.34) If it is deemed necessary to include the vulnerable populations, it is recommended to include them in PCTs if additional protection is secured.55)

Preparing for dissemination and implementation

When reporting the findings of PCTs to the stakeholders and public, transparency and applicability are essential in making informed decisions about the practice based on the finding. In 2008, the CONsolidated Standards of Reporting Trials (CONSORT) statement14) was published and used to assess the validity and reliability of the results and recommended the performance of PCTs. An extension of the CONSORT statement supports extending the following eight CONSORT checklist items for reporting of PCTs: background, participants, intervention, outcomes, sample size, blinding, participant flow, and generalizability of the findings. The dissemination of results in PCTs conceptualizes and measures the implementation outcomes in the real-world setting. An example of dissemination strategies is the use of a framework such as the Reach, Effectiveness, Adoption, Implementation, and Maintenance (RE-AIM) Framework, which can emphasize the effectiveness and reach of PCTs.

EXAMPLES OF PCTs

As identified above, PCTs are performed to verify effectiveness in the real-world setting and provide important answers to clinical questions. Since the concept of PCTs was first proposed, some clinical trials labeled pragmatic have been conducted, but most studies that adequately reflect the actual characteristics of pragmatism were conducted just recently. The following five examples are some of the PCTs that have been implemented (Table 3).

First, the Clinical Randomisation of an Antifibrinolytic in Significant Haemorrhage (CRASH-2) study56) was a randomized placebo-controlled trial aimed to assess the effectiveness of tranexamic acid in bleeding trauma patients in 274 hospitals from 40 countries. The results of CRASH-2 showed that tranexamic acid treatment within 3 hours of injury significantly reduced the risk of death. Based on the results of CRASH-2, the early administration of tranexamic acid in patients with trauma has been stated in several guidelines.57-60) The Post-Myocardial Infarction Free Rx and Economic Evaluation trial61) was a simple cluster-randomized and controlled policy study. This study was developed to assess the effect of providing full prescription drug coverage for statins, β-blockers, angiotensinconverting enzyme inhibitors, and angiotensin II receptor blockers to patients immediately after discharge from the hospital. This is the first randomized study to evaluate the impact of reducing cost-sharing for essential cardiac medications on clinical and economical outcomes. Thrombus aspiration in patients with an ST-elevation myocardial infarction in Scandinavia, Sweden (Thrombus Aspiration in ST-Elevation Myocardial Infarction in Scandinavia; TASTE Trial)62) was performed to test whether thrombus aspiration reduces mortality in patients with an ST-segment elevation myocardial infarction (STEMI) compared with standard care before percutaneous coronary intervention (PCI). However, thrombus aspiration before PCI did not reduce the 30-day mortality in patients with STEMI compared with PCI alone. This trial introduced randomization in the national health registry, thus combining the strengths of a randomized assignment and the benefits of a large-scale registry. Salford Lung Study (SLS) was the first PCT conducted prior to the drug approval.63) SLS assessed the effectiveness and safety of the combination of inhaled corticosteroids (fluticasone furoate) and the long-acting β2-agonist (vilanterol) compared with those of existing chronic obstructive pulmonary disease (COPD) maintenance therapies in patients with COPD in Salford, UK. Results showed that a once a day treatment of a combination of fluticasone furoate and vilanterol lowers the rate of exacerbations in COPD patients without any risk of serious adverse events.64) The High-Sensitivity Troponin in the Evaluation of patients with Acute Coronary Syndrome (High-STEACS) trial in Scotland evaluated whether a high-sensitivity cardiac troponin I (hs-cTnI) assay is more effective in reducing myocardial infarction or cardiovascular death at 1 year than a standard troponin assay. This is the first PCT to evaluate the effectiveness of the hs-cTnI assay in patients with a suggested acute coronary syndrome.

The examples of trials mentioned above have various implications to confirm the effectiveness in a real-world setting using a large population from various countries and to expand the data sources that can be used for PCTs. They also reported that the results of PCTs possibly influenced the change in the clinical guidelines, and that PCTs could be used for premarket approval purposes. These trials would be useful examples when conducting PCTs to obtain the RWE of treatments in various countries.

STRENGTH AND LIMITATION OF PCTs

The great advantage of PCTs is that they are designed to test practicability and effectiveness in the actual day-to-day care setting.65) Therefore, it is possible to create a basis to assist in the selection of treatment for patients or health care providers16) and to provide answers to decision makers such as health care policymakers to allocate resources and personnel efficiently.7) It is highly likely that physicians and patients will participate in clinical trials owing to the flexibility of patient treatment and the absence of placebo treatment.16)

The main limitation is that PCTs can reduce the internal validity; however, the external validity can be obtained.65) Therefore, PCTs should use a large sample size and have a long-term follow-up period; this can create a huge cost burden associated with increasing resources.7)16) Because PCTs compare the effectiveness of many interventions in clinical practice, it is possible to understand the overall performance, but it is difficult to determine the exact magnitude of the effect caused by a particular component.7) In other words, the fact that the result of PCTs in treatment is effective in routine care does not indicate the exact amount of effectiveness in an actual individual case.7) In the case of clinical trials that evaluate new indications or doses of treatment before the drug is launched, they cannot be considered as PCTs because they are required to comply with clinical trial regulations, not with routine care.8)

CONCLUSIONS

PCTs provide an important basis for decision-making in the real-world clinical practice and policy settings while maintaining the strengths of classical RCTs if they are systematically designed and performed properly. The recent growing interest and importance of RWE have also increased the researchers' attention to the performance of PCTs. To conduct PCTs, it is better to check whether PCTs are feasible and then perform PCTs when they are thought to be appropriate means to obtain RWE while retaining the quality of clinical trials. Although only a few PCTs have been conducted in Korea, the interest in conducting PCTs is increasing lately. In conclusion, to produce RWE related to clinical and policy decisions, academia, physicians, policy makers, and patients must understand the concepts and benefits of PCTs, and they must be encouraged to use PCTs.

Notes

Conflict of Interest

The authors have no financial conflicts of interest.

Author Contributions

Conceptualization: Jeong NY, Choi NK; Investigation: Jeong NY, Kim SH, Lim E, Choi NK; Supervision: Choi NK; Writing - original draft: Jeong NY, Kim SH, Lim E; Writing - review & editing: Jeong NY, Kim SH, Lim E, Choi NK.

REFERENCES

1. Silverman SL. From randomized controlled trials to observational studies. Am J Med. 2009; 122:114–20.

2. Treweek S, Zwarenstein M. Making trials matter: pragmatic and explanatory trials and the problem of applicability. Trials. 2009; 10:37.

3. Gill J, Prasad V. Improving observational studies in the era of big data. Lancet. 2018; 392:716–7.

4. Benson K, Hartz AJ. A comparison of observational studies and randomized, controlled trials. N Engl J Med. 2000; 342:1878–86.

5. Grimes DA, Schulz KF. Bias and causal associations in observational research. Lancet. 2002; 359:248–52.

6. Zuidgeest MG, Goetz I, Groenwold RH, Irving E, van Thiel GJ, Grobbee DE; GetReal Work Package 3. Series: pragmatic trials and real world evidence: Paper 1. Introduction. J Clin Epidemiol. 2017; 88:7–13.

7. Patsopoulos NA. A pragmatic view on pragmatic trials. Dialogues Clin Neurosci. 2011; 13:217–24.

8. Dal-Ré R, Janiaud P, Ioannidis JP. Real-world evidence: how pragmatic are randomized controlled trials labeled as pragmatic? BMC Med. 2018; 16:49.

9. Schwartz D, Lellouch J. Explanatory and pragmatic attitudes in therapeutical trials. J Chronic Dis. 1967; 20:637–48.

10. Tunis SR, Stryer DB, Clancy CM. Practical clinical trials: increasing the value of clinical research for decision making in clinical and health policy. JAMA. 2003; 290:1624–32.

11. Califf RM, Sugarman J. Exploring the ethical and regulatory issues in pragmatic clinical trials. Clin Trials. 2015; 12:436–41.

12. Gamerman V, Cai T; Elsäßer A. Pragmatic randomized clinical trials: best practices and statistical guidance. Health Serv Outcomes Res Methodol. 2019; 19:23–35.

13. Zwarenstein M, Treweek S. What kind of randomized trials do we need? J Clin Epidemiol. 2009; 62:461–3.

14. Zwarenstein M, Treweek S, Gagnier JJ, Altman DG, Tunis S, Haynes B, Oxman AD, Moher D, CONSORT group; Pragmatic Trials in Healthcare (Practihc) group. Improving the reporting of pragmatic trials: an extension of the CONSORT statement. BMJ. 2008; 337:a2390.

15. Coats TJ. Future research in emergency medicine: explanation or pragmatism? Large or small? Simple or complex? Emerg Med J. 2011; 28:1004–7.

16. MacPherson H. Pragmatic clinical trials. Complement Ther Med. 2004; 12:136–40.

17. Borgerson K. Are explanatory trials ethical? Shifting the burden of justification in clinical trial design. Theor Med Bioeth. 2013; 34:293–308.

18. Loudon K, Treweek S, Sullivan F, Donnan P, Thorpe KE, Zwarenstein M. The PRECIS-2 tool: designing trials that are fit for purpose. BMJ. 2015; 350:h2147.

19. Thorpe KE, Zwarenstein M, Oxman AD, Treweek S, Furberg CD, Altman DG, Tunis S, Bergel E, Harvey I, Magid DJ, Chalkidou K. A pragmatic-explanatory continuum indicator summary (PRECIS): a tool to help trial designers. J Clin Epidemiol. 2009; 62:464–75.

20. Bevan A, Biedenbach P, Berger A. Pragmatic randomized trials: considerations for design and implementation [Internet]. Bethesda, MD: Evidera;2019. [cited 2020 May]. Available from https://www.evidera.com/pragmatic-randomized-trials-considerations-for-design-and-implementation/.

21. Kalkman S, van Thiel G, van der Graaf R, Zuidgeest M, Goetz I, Grobbee D, van Delden J. The social value of pragmatic trials. Bioethics. 2017; 31:136–43.

22. Concannon TW, Meissner P, Grunbaum JA, McElwee N, Guise JM, Santa J, Conway PH, Daudelin D, Morrato EH, Leslie LK. A new taxonomy for stakeholder engagement in patient-centered outcomes research. J Gen Intern Med. 2012; 27:985–91.

23. Johnson KE, Tachibana C, Coronado GD, Dember LM, Glasgow RE, Huang SS, Martin PJ, Richards J, Rosenthal G, Septimus E, Simon GE, Solberg L, Suls J, Thompson E, Larson EB. A guide to research partnerships for pragmatic clinical trials. BMJ. 2014; 349:g6826.

24. Fraser J, Moloney R, Tambor E, Tuzzio L. Building partnerships to ensure a successful trial: deciding who to engage. In: Rethinking Clinical Trials: A Living Textbook of Pragmatic Clinical Trials [Internet]. Bethesda, MD: NIH Health Care Systems Research Collaboratory;2020. [cited 2020 May 30]. Available from https://rethinkingclinicaltrials.org/chapters/engaging-stakeholders/deciding-who-to-engagev2/.

25. Chambers D, Coronado G, Green B, Jarvik J, Septimus E, Tuzzio L, Zatzick D. Designing with implementation and dissemination in mind: key considerations. In: Rethinking Clinical Trials: A Living Textbook of Pragmatic Clinical Trials [Internet]. Bethesda, MD: NIH Health Care Systems Research Collaboratory;2018. [cited 2020 May 30]. Available from https://rethinkingclinicaltrials.org/chapters/design/designing-implementation-dissemination-mind-top/designing-with-implementation-and-dissemination-in-mind-2/.

26. Simon G, Ramsberg J, Coronado G, Septimus E, DeBar L, Vazquez M, Dember LM, Vollmer WM, Green B, Zatzick D, Huang SS, Hernandez A, Jarvik J, Platt R, Mor V. Data sharing and embedded research: introduction. In: Rethinking Clinical Trials: A Living Textbook of Pragmatic Clinical Trials [Internet]. Bethesda, MD: NIH Health Care Systems Research Collaboratory;2019. [cited 2020 May 30]. Available from https://rethinkingclinicaltrials.org/chapters/dissemination/data-share-top/data-sharing-andembedded-research-introduction/.

27. Welsing PM, Oude Rengerink K, Collier S, Eckert L, van Smeden M, Ciaglia A, Nachbaur G, Trelle S, Taylor AJ, Egger M, Goetz I; Work Package 3 of the GetReal Consortium. Series: pragmatic trials and real world evidence: Paper 6. Outcome measures in the real world. J Clin Epidemiol. 2017; 90:99–107.

28. Curtis L, Hernandez A, Weinfurt K. Choosing and specifying endpoints and outcomes: meaningful endpoints. In: Rethinking Clinical Trials: A Living Textbook of Pragmatic Clinical Trials [Internet]. Bethesda, MD: NIH Health Care Systems Research Collaboratory;2020. [cited 2020 May 30]. Available from https://rethinkingclinicaltrials.org/chapters/design/choosing-specifying-end-points-outcomes/meaningful-endpoints/.

29. Murad MH, Shah ND, Van Houten HK, Ziegenfuss JY, Deming JR, Beebe TJ, Smith SA, Guyatt GH, Montori VM. Individuals with diabetes preferred that future trials use patient-important outcomes and provide pragmatic inferences. J Clin Epidemiol. 2011; 64:743–8.

30. Basch E. New frontiers in patient-reported outcomes: adverse event reporting, comparative effectiveness, and quality assessment. Annu Rev Med. 2014; 65:307–17.

31. Walton MK, Powers JH 3rd, Hobart J, Patrick D, Marquis P, Vamvakas S, Isaac M, Molsen E, Cano S, Burke LB; International Society for Pharmacoeconomics and Outcomes Research Task Force for Clinical Outcomes Assessment. Clinical outcome assessments: conceptual foundation-report of the ISPOR Clinical Outcomes Assessment - Emerging Good Practices for Outcomes Research Task Force. Value Health. 2015; 18:741–52.

32. Worsley SD, Oude Rengerink K, Irving E, Lejeune S, Mol K, Collier S, Groenwold RH, Enters-Weijnen C, Egger M, Rhodes T; GetReal Work Package 3. Series: pragmatic trials and real world evidence: Paper 2. Setting, sites, and investigator selection. J Clin Epidemiol. 2017; 88:14–20.

33. Bulpitt CJ. Randomised Controlled Clinical Trials. New York, NY: Springer Science & Business Media;2012.

34. Oude Rengerink K, Kalkman S, Collier S, Ciaglia A, Worsley SD, Lightbourne A, Eckert L, Groenwold RH, Grobbee DE, Irving EA; Work Package 3 of the GetReal consortium. Series: pragmatic trials and real world evidence: Paper 3. Patient selection challenges and consequences. J Clin Epidemiol. 2017; 89:173–80.

35. Cook AJ, Delong E, Murray DM, Vollmer WM, Heagerty PJ. Statistical lessons learned for designing cluster randomized pragmatic clinical trials from the NIH Health Care Systems Collaboratory Biostatistics and Design Core. Clin Trials. 2016; 13:504–12.

36. Godwin M, Ruhland L, Casson I, MacDonald S, Delva D, Birtwhistle R, Lam M, Seguin R. Pragmatic controlled clinical trials in primary care: the struggle between external and internal validity. BMC Med Res Methodol. 2003; 3:28.

37. Fransen GA, van Marrewijk CJ, Mujakovic S, Muris JW, Laheij RJ, Numans ME, de Wit NJ, Samsom M, Jansen JB, Knottnerus JA. Pragmatic trials in primary care. Methodological challenges and solutions demonstrated by the DIAMOND-study. BMC Med Res Methodol. 2007; 7:16.

38. Jiang D, Chen K, Mukhopadhyay S, Katta N, Zhang L. Review of Statistical Issues in Pragmatic Clinical Trials in Current Drug Development Environment. In: Contemporary Biostatistics with Biopharmaceutical Applications. Cham, Switzerland: Springer;2019. p. 285–98.

39. Gupta SK. Intention-to-treat concept: a review. Perspect Clin Res. 2011; 2:109–12.

40. Sugarman J, Califf RM. Ethics and regulatory complexities for pragmatic clinical trials. JAMA. 2014; 311:2381–2.

41. McKinney RE Jr, Beskow LM, Ford DE, Lantos JD, McCall J, Patrick-Lake B, Pletcher MJ, Rath B, Schmidt H, Weinfurt K. Use of altered informed consent in pragmatic clinical research. Clin Trials. 2015; 12:494–502.

42. Kim SY, Miller FG. Informed consent for pragmatic trials--the integrated consent model. N Engl J Med. 2014; 370:769–72.

43. Wendler D. “Targeted” consent for pragmatic clinical trials. J Gen Intern Med. 2015; 30:679–82.

44. Faden R, Kass N, Whicher D, Stewart W, Tunis S. Ethics and informed consent for comparative effectiveness research with prospective electronic clinical data. Med Care. 2013; 51:S53–7.

45. Faden RR, Beauchamp TL, Kass NE. Informed consent, comparative effectiveness, and learning health care. N Engl J Med. 2014; 370:766–8.

46. D'Agostino RB, Sullivan LM, Massaro J. Wiley Encyclopedia of Clinical Trials. Hoboken, NJ: John Wiley & Sons, Inc;2007.

47. Ellenberg S, Sugarman J, Zatzick D. Data and safety monitoring: which PCTs should have a DMC? In: Rethinking Clinical Trials: A Living Textbook of Pragmatic Clinical Trials [Internet]. Bethesda, MD: NIH Health Care Systems Research Collaboratory;2020. [cited 2020 May 30]. Available from: https://rethinkingclinicaltrials.org/chapters/design/planning-data-safety-monitoring/which-pcts-should-have-a-dmc/.

48. Food and Drug Administration. FDA Guidance for Clinical Trial Sponsors On the Establishment and Operation of Clinical Trial Data Monitoring Committees. Rockville, MD: Food and Drug Administration;2006.

49. DeBar LL, Jarvik JG, Tuzzio L, Vazquez MA. Assessing feasibility: introduction. In: Rethinking Clinical Trials: A Living Textbook of Pragmatic Clinical Trials [Internet]. Bethesda, MD: NIH Health Care Systems Research Collaboratory;2018. [cited 2020 May 30]. Available from https://rethinkingclinicaltrials.org/chapters/conduct/assessing-feasibility/assessing-feasibility-introduction/.

50. DeBar LL, Jarvik JG, Tuzzio L, Vazquez MA. Assessing feasibility: developing the trial documentation. In: Rethinking Clinical Trials: A Living Textbook of Pragmatic Clinical Trials [Internet]. Bethesda, MD: NIH Health Care Systems Research Collaboratory;2018. [cited 2020 May 30]. Available from https:// rethinkingclinicaltrials.org/chapters/conduct/assessing-feasibility/developing-the-trial-documentation/.

51. DeBar LL, Jarvik JG, Tuzzio L, Vazquez MA. Assessing feasibility: pilot testing. In: Rethinking Clinical Trials: A Living Textbook of Pragmatic Clinical Trials [Internet]. Bethesda, MD: NIH Health Care Systems Research Collaboratory;2020. [cited 2020 May 30]. Available from https://rethinkingclinicaltrials.org/chapters/conduct/assessing-feasibility/pilot-testing/.

52. DeBar LL, Jarvik JG, Tuzzio L, Vazquez MA. Study startup: introduction. In: Rethinking Clinical Trials: A Living Textbook of Pragmatic Clinical Trials [Internet]. Bethesda, MD: NIH Health Care Systems Research Collaboratory;2020. [cited 2020 May 30]. Available from https://rethinkingclinicaltrials.org/chapters/conduct/startup/startup-introduction/.

53. DeBar L, Jarvik J, Tuzzio L, Vazquez M. Study Startup: Implementation Readiness Checklist. In: Rethinking Clinical Trials: A Living Textbook of Pragmatic Clinical Trials [Internet]. Bethesda, MD: NIH Health Care Systems Research Collaboratory;2020. [cited 2020 May 30]. Available from https://rethinkingclinicaltrials.org/chapters/conduct/startup/startup-implementation/.

54. Cook DJ, Ferguson ND, Hand L, Austin P, Zhou Q, Adhikari NK, Danesh V, Arabi Y, Matte AL, Clarke FE, Mehta S, Smith O, Wise MP, Friedrich JO, Keenan SP, Hanna S, Meade MO; OSCILLation for ARDS Treated Early Investigators; Canadian Critical Care Trials Group. Coenrollment in a randomized trial of highfrequency oscillation: prevalence, patterns, predictors, and outcomes. Crit Care Med. 2015; 43:328–38.

55. Welch MJ, Lally R, Miller JE, Pittman S, Brodsky L, Caplan AL, Uhlenbrauck G, Louzao DM, Fischer JH, Wilfond B. The ethics and regulatory landscape of including vulnerable populations in pragmatic clinical trials. Clin Trials. 2015; 12:503–10.

56. CRASH-2 trial collaborators, Shakur H, Roberts I, Bautista R, Caballero J, Coats T, Dewan Y, El-Sayed H, Gogichaishvili T, Gupta S, Herrera J, Hunt B, Iribhogbe P, Izurieta M, Khamis H, Komolafe E, Marrero MA, Mejía-Mantilla J, Miranda J, Morales C, Olaomi O, Olldashi F, Perel P, Peto R, Ramana PV, Ravi RR, Yutthakasemsunt S. Effects of tranexamic acid on death, vascular occlusive events, and blood transfusion in trauma patients with significant haemorrhage (CRASH-2): a randomised, placebo-controlled trial. Lancet. 2010; 376:23–32.

57. Wada H, Thachil J, Di Nisio M, Mathew P, Kurosawa S, Gando S, Kim HK, Nielsen JD, Dempfle CE, Levi M, Toh CH; The Scientific Standardization Committee on DIC of the International Society on Thrombosis Haemostasis. Guidance for diagnosis and treatment of DIC from harmonization of the recommendations from three guidelines. J Thromb Haemost. 2013; 11:761–7.

58. Hunt BJ, Allard S, Keeling D, Norfolk D, Stanworth SJ, Pendry K; British Committee for Standards in Haematology. A practical guideline for the haematological management of major haemorrhage. Br J Haematol. 2015; 170:788–803.

59. Rossaint R, Bouillon B, Cerny V, Coats TJ, Duranteau J, Fernández-Mondéjar E, Filipescu D, Hunt BJ, Komadina R, Nardi G, Neugebauer EA, Ozier Y, Riddez L, Schultz A, Vincent JL, Spahn DR. The European guideline on management of major bleeding and coagulopathy following trauma: fourth edition. Crit Care. 2016; 20:100.

60. National Clinical Guideline Centre. Major Trauma: Assessment and Initial Management. London: National Institute for Health and Care Excellence;2016.

61. Choudhry NK, Avorn J, Glynn RJ, Antman EM, Schneeweiss S, Toscano M, Reisman L, Fernandes J, Spettell C, Lee JL, Levin R, Brennan T, Shrank WH; Post-Myocardial Infarction Free Rx Event and Economic Evaluation (MI FREEE) Trial. Full coverage for preventive medications after myocardial infarction. N Engl J Med. 2011; 365:2088–97.

62. Fröbert O, Lagerqvist B, Olivecrona GK, Omerovic E, Gudnason T, Maeng M, Aasa M, Angerås O, Calais F, Danielewicz M, Erlinge D, Hellsten L, Jensen U, Johansson AC, Kåregren A, Nilsson J, Robertson L, Sandhall L, Sjögren I, Ostlund O, Harnek J, James SK, TASTE Trial. Thrombus aspiration during STsegment elevation myocardial infarction. N Engl J Med. 2013; 369:1587–97.

63. Bakerly ND, Woodcock A, New JP, Gibson JM, Wu W, Leather D, Vestbo J. The Salford Lung Study protocol: a pragmatic, randomised phase III real-world effectiveness trial in chronic obstructive pulmonary disease. Respir Res. 2015; 16:101.

64. Vestbo J, Leather D, Diar Bakerly N, New J, Gibson JM, McCorkindale S, Collier S, Crawford J, Frith L, Harvey C, Svedsater H, Woodcock A; Salford Lung Study Investigators. Effectiveness of fluticasone furoate-vilanterol for COPD in clinical practice. N Engl J Med. 2016; 375:1253–60.

65. Kumar A, Rana V. Pragmatic trials: importance and impact. J Endocrinol Thyroid Res. 2017; 2:555594.

66. Shah AS, Anand A, Strachan FE, Ferry AV, Lee KK, Chapman AR, Sandeman D, Stables CL, Adamson PD, Andrews JP, Anwar MS, Hung J, Moss AJ, O'Brien R, Berry C, Findlay I, Walker S, Cruickshank A, Reid A, Gray A, Collinson PO, Apple FS, McAllister DA, Maguire D, Fox KA, Newby DE, Tuck C, Harkess R, Parker RA, Keerie C, Weir CJ; Mills NLHigh-STEACS Investigators. High-sensitivity troponin in the evaluation of patients with suspected acute coronary syndrome: a stepped-wedge, cluster-randomised controlled trial. Lancet. 2018; 392:919–28.

Figure 1.

Relationships among the explanatory clinical trials, observational study, and pragmatic clinical trials, including hypothetical PRECIS-2 diagrams. PRECIS-2 diagrams in the figure is referred to Bevan et al.20)

PRECIS = PRagmatic Explanatory Continuum Indicator Summary.

Table 1.

Comparison of the attributes that distinguish ECTs and PCTs

Feature	ECTs	PCTs
Question	Efficacy: does the intervention work under ideal setting?	Effectiveness: does the intervention benefit when used in routine practice?
Inclusion criteria of participants	Strict criteria to exclude high-risk or poorly adherent participants	Broader criteria to include various participants
Setting	Highly controlled experimental setting	Normal practice setting
Intervention	Thorough delivery and monitoring of intervention	Flexible delivery and monitoring of intervention
Comparator	Generally placebo control	Routine clinical treatment, mostly not placebo control
Outcomes	A restricted set of events or surrogated outcomes	A broad set of events or determined in the routine course of clinical practice
Sample size	Comparatively small	Comparatively large
Relevance to practice	Low relevance to practice	High relevance to practice
Follow-up period	Relatively short-term follow-up	Relatively long-term follow-up

ECT = explanatory clinical trial; PCT = pragmatic clinical trial.

Table 2.

Outline and pragmatic approach about PRECIS-2 domain

Domain	Outline	Highly pragmatic approach
Eligibility	To what extent are the participants in the trial similar to those who would receive this intervention if it was part of usual care?	Anyone covered elderly, children, people with comorbidities was included.
Recruitment	How much extra effort is made to recruit participants over and above what would be used in the usual care setting to engage with patients?	People who visit the hospital with the condition of interested were included without overt recruitment effort.
Setting	How different are the settings of the trial from the usual care setting?	Trials were performed in an identical setting to clinical practice that applies the results.
Organization	How different are the resources, provider expertise, and the organization of care delivery in the intervention arm of the trial from those available in usual care?	Trials slotted the intervention into the usual healthcare organization, trying to use of the existing healthcare staff and resources.
Flexibility (delivery)	How different is the flexibility in how the intervention is delivered and the flexibility anticipated in usual care?	Methodology of how to deliver the intervention was not strictly prescriptive, and it did not dictate which other interventions were permitted.
Flexibility (adherence)	How different is the flexibility in how participants are monitored and encouraged to adhere to the intervention from the flexibility anticipated in usual care?	Trials allowed the full flexibility in how end-user recipients engage with the intervention with no specific measures to enforce engagement or compliance.
Follow-up	How different is the intensity of measurement and follow-up of participants in the trial from the typical follow-up in usual care?	Trials had no more follow-up than usual care and collected minimal additional data.
Primary outcome	To what extent is the trial's primary outcome directly relevant to participants?	Trials chose obviously important outcomes to patients and measured them in a way that was the same or similar to the way they are measured in practice.
Primary analysis	To what extent are all data included in the analysis of the primary outcome?	Trials used an intention-to-treat analysis about all available data.

Information in the table is referred to Loudon et al.18)

PRECIS = PRagmatic Explanatory Continuum Indicator Summary.

Table 3.

Exemplar of published PCTs and their main features

Trial	Year	Ref.	Question	Inclusion criteria of participants (No. of participants; No. of sites)	Intervention (No. of participants) vs. Comparator (No. of participants)	Randomization	Study period^† (follow-up period)	Primary outcomes	Results
CRASH-2	2020	56	Whether the early administration of tranexamic acid reduces death, vascular occlusive events, and the receipt of blood transfusion	Adult trauma patients with significant haemorrhage and who were within 8 hours of injury (n=20,211; n=274)	Tranexamic acid (n=10,060) vs. Placebo (n=10,067)	Central randomization at the level of individuals	May 2005 to Mar 2010 (4 weeks)	Death in hospital	Tranexamic acid safely reduced the risk of death in bleeding trauma patients
Post-MI FREEE	2011	61	Whether full prescription drug coverage for statins, β-blockers, ACE inhibitors, and ARBs is more effective to patients after MI	Adults discharged alive from hospital after MI who received health services and prescription drug benefits through Aetna, Inc. (n=5,855; N/A^*)	Full prescription coverage (n=2,845) vs. Usual prescription coverage (n=3,010)	Cluster randomization at the level of the plan sponsor	Nov 2007 to Nov 2010 (≥1 year)	First vascular event or revascularization	The removal of copayments for drugs prescribed after being discharged due to MI did not significantly reduce rates of the primary outcome
TASTE	2013	62	Whether thrombus aspiration before PCI reduces mortality than usual care in patients with STEMI	Patients in a registry for coronary angiography and angioplasty (n=7,244; n=31)	Thrombus aspiration before PCI (n=3,621) vs. Usual care (n=3,623)	Registry based randomization	Jul 2010 to Aug 2013 (30 days)	All-cause death	Thrombus aspiration before PCI did not reduce 30-day mortality in patients with STEMI compared to PCI alone
SLS	2016	64	Whether the effectiveness and safety of the once-daily inhaled combination of fluticasone furoate–vilanterol is better than existing maintenance therapy	Adults (≥40) who received diagnosis of COPD and who had ≥1 COPD exacerbations in the previous 3 years in 75 general practices (n=2,799; n=75)	Fluticasone furoate- vilanterol (n=2,799) vs. Usual care (n=1,403)	Central randomization at the level of individuals	Mar 2012 to Nov 2015 (1 year)	Mean annual rate of exacerbations	Once-daily combined treatment of fluticasone furoate and vilanterol decreased the rate of exacerbations without a further risk of serious adverse events than usual care
High-STEACS	2018	66	Whether the use of hs-cTnI assay reduces MI or cardiovascular death compared with standard troponin assay	Patients with acute coronary syndrome suspects in 10 Scotland hospitals (n=48,282; n=10)	Reclassified as MI by hs-cTnI assay (n=1,771) vs. Classified as MI by cardiac troponin I assay (n=8,589)	Stepped wedge cluster randomization at the level of the hospital site	Jun 2013 to Mar 2017 (1 year)	Subsequent MI or cardiovascular death	High-sensitivity assay did not reduce in MI or cardiovascular death within 1 year

ACE = angiotensin-converting enzyme; ARBs = angiotensin II receptor blockers; CRASH-2 = Clinical Randomisation of an Antifibrinolytic in Significant Haemorrhage; COPD = chronic obstructive pulmonary disease; High-STEACS = High-Sensitivity Troponin in the Evaluation of patients with Acute Coronary Syndrome; hs-cTnI = high-sensitivity cardiac troponin I; MI = myocardial infarction; PCI = percutaneous coronary intervention; PCT = randomized controlled trial; Post-MI FREEE = The Post-Myocardial Infarction Free Rx and Economic Evaluation; SLS = Salford Lung Study; STEMI = ST-segment elevation myocardial infarction; TASTE = Thrombus Aspiration in ST-Elevation Myocardial Infarction in Scandinavia.

^* N/A (not available) means that accurate information was not available in the reference literature.

^† Study periods of each trial were referred to ClinicalTrials.gov.

TOOLS

Similar articles