The term RM refers to a wide variety of material types. These materials can be certified or non-certified, qualitative or quantitative, and can be characterized to different degrees. The International Organization for Standardization (ISO) defines a RM as "a material that is homogeneous and stable with respect to one or more specified properties, which has been established to fit for its intended use in a measurement process" (ISO Guide 30:2015) [9] (Table 1).

Certified RMs (CRMs) are characterized by at least one metrologically valid procedure for at least one specified property with a stated uncertainty. They come with a certificate providing the value of the specified property, its associated uncertainty, and a statement of metrological traceability. CRMs are characterized for composition and are traceable to the International System of Units (SI) without reference to a calibrator, and are produced by a variety of accredited organizations. The NIST releases CRMs under the trademark "standard RMs (SRMs)" [10] and must meet additional NIST-specific criteria. European RMs (ERMs) are CRMs produced by the European RM consortium [11]. Other RMs, including the whole genome NIST RM 8398, are characterized for homogeneity and stability, but have no certified values. The World Health Organization (WHO) disseminates a variety of RMs that are widely used to harmonize clinical assay testing [12]. WHO standards are typically prepared in a natural matrix, such as plasma or serum, facilitating compatibility between assays. The values assigned to WHO International Standards are determined by the consensus of a number of laboratories that test the samples, rather than a reference method [7]. Other commonly used RMs include characterized genomic DNA from cell lines or residual patient DNA. These RMs, while often less formally characterized and distributed than other RM types, are useful for assuring the quality of laboratory tests when more characterized materials are not available.

A number of "higher order" RMs are available for molecular genetic testing [1314]. These include CRMs, SRMs, and WHO standards for Factor V Leiden; Fragile X; BCR-ABL1; prothrombin; Huntington's disease; mitochondrial DNA; the human genome; blood group genotyping; Hepatitis A, B, and C; cytomegalovirus (CMV); HIV; human papilloma virus (HPV); and a number of bacteria and yeast. In May 2015, NIST released RM 8398, Human DNA for Whole-Genome Variant Assessment [8]. This RM has been characterized for homogeneity and stability, and has reference values for single nucleotide polymorphisms (SNP), small insertions, and deletions (indels), and homozygous reference genotypes for approximately 77% of the human genome [15]. Variants are reported with respect to the GRCh37 Human Reference Assembly. However, CRMs, SRMs, and WHO standards are relatively expensive and are only available in small quantities. They are intended as primary standards that can be used to produce secondary standards traceable to the RM, which are then used as daily controls.

Although "higher order" RMs are available for a small number of tests and often in small amounts, many other RMs are available in larger quantities and are appropriate for use on a daily basis as QC materials and for test development and validation [67]. These materials include commercially available controls, synthetic controls, and characterized genomic DNA and cell lines. Some of these materials are well-characterized, produced in accordance with good manufacturing practices and/or ISO guidelines, and/or approved or certified by national standards organizations (eg. Conformité Européenne [CE]-marked, Food and Drug Administration [FDA]-cleared). Other less-characterized materials are also sold as reference standards. Some materials, such as cell lines and genomic DNA, may be minimally characterized or uncharacterized and require further validation before use in clinical laboratories.

Clinical laboratories use RMs for a variety of purposes, including test development, validation, QC, and proficiency testing. Use of these materials is essential for assuring the quality of clinical tests, and their use for assay validation and quality assurance is required by regulatory and accreditation bodies and is generally recommended by professional guidance [6161718192021]. Although clinical genetic tests are available for over 3,000 conditions [22], RMs are not available for the vast majority of these assays. In the absence of publicly available RMs, clinical laboratories often use DNA derived from leftover patient specimens as controls. These RM specimens are not easily available, nor are they renewable, and they must be validated by multiple methods prior to use.

The US Centers for Disease Control and Prevention (CDC), in partnership with the genetic testing community, established the Genetic Testing Reference Materials Coordination Program (Get-RM) [23] in 2004 to address the urgent need for characterized genomic DNA RMs. The GeT-RM program is coordinated by the CDC, but all the decisions about RM priorities, specimen collection, material development, and characterization occur through voluntary collaborations with laboratories in the clinical genetic testing community. Cell lines with confirmed genotypes are the preferred type of control for DNA-based genetic testing as they closely resemble an actual patient specimen. Thus, the GeT-RM has focused on characterizing genomic DNA from publicly available cell lines.

GeT-RM RM characterization projects typically begin with identifying the need for a RM to carry out tests for a single gene disorder (eg. fragile X) or a gene panel (eg. Ashkenazi Jewish [AJ]) (Fig. 1). Decisions about variants to be included among the characterized genomic DNA materials are made on the basis of consultations with clinical laboratories, commercial test developers, researchers, and cell repositories. Panels of genomic DNA samples are assembled to include many or all common variants or variant types known to cause the disorder(s), important combinations of variants, heterozygotes, homozygotes, compound heterozygotes, samples from males and females (X-linked disorders), non-pathogenic polymorphisms that affect assay performance, and variants at or near clinical decision points. Samples are also selected on the basis of the consideration of the assay technologies used [23].

Once a list of desired samples is compiled, cell lines containing these variants are selected from the Coriell Cell Repositories [24]. If the repository does not have cell lines with the desired variants, appropriate cell lines can be found or created. Cell lines positive for the requested variants can be transferred from research laboratories to the Coriell repository. In addition, the GeT-RM, in partnership with the Coriell Repositories, has created over 100 cell lines from blood samples collected from consenting patients who were contacted through patient advocacy organizations or clinical research projects. These newly created cell lines provided variants that were not already available from the Coriell Repository.

Once the selected cell lines have been identified or created, genomic DNA samples are sent to three or more volunteer laboratories for characterization. The laboratories are selected on the basis of the assay technology used to detect the variants and their willingness to test the samples. A variety of assay technologies and methods are used to characterize the samples to assure that the expected variant can be detected, and that the samples are commutable for use with current clinical assays. Results are sent to the GeT-RM, and a consensus genotype is assigned to each sample. The GeT-RM program publishes characterization studies [25262728293031323334] and the data are also available on the GeT-RM [23] and Coriell [24] websites.

Since 2004, the GeT-RM program has characterized more than 4,600 loci in over 400 cell line-based genomic DNA RMs for a number of genetic disorders including: fragile X [25], disorders on the AJ Panel [29] (Bloom syndrome, Canavan disease, Fanconi anemia, familial dysautonomia, Gaucher disease, mucolipidosis IV, Niemann Pick disease and Tay Sachs disease), cystic fibrosis [26], Huntington disease [27], methylenetetrahydrofolate reductase (MTHFR)-related homocysteinemia, α1-antitrypsin deficiency, multiple endocrine neoplasia, BRCA1- and 2-related cancers [28], Duchenne muscular dystrophy [30], myotonic dystrophy [31], Rett syndrome [32], and two large-scale DNA studies from 244 cell lines for a number of polymorphisms in 28 pharmacogenetic loci [3334]. Each of these genomic DNA materials was tested in 3-10 clinical genetic laboratories by using a variety of genetic assays, including DNA sequence analysis. All cell lines and genomic DNA samples characterized by GeT-RM are publicly available from the Coriell Cell Repositories [24]. Information about genomic DNA samples characterized by the GeT-RM, as well as other information about other available RMs, is available from the GeT-RM website [23].

NIST hosted the first meetings of the GIAB Consortium in 2012 to help develop RMs, reference data, and reference methods for assessing NGS performance. The GIAB RM Selection and Design Working Group recommended human cell lines for NIST to develop into RMs. The consortium selected NA12878 from the Coriell Cell Repositories as a pilot sample. This genome was widely used in public studies and was extensively characterized by the 1000 Genomes Project, sequencing technology developers, and other laboratories. The publicly available data for this genome was used to develop methods for integrating multiple datasets to establish high-confidence SNP, indels, and homozygous reference genotype calls [15]. In addition, NIST and GIAB have developed high-confidence calls for large deletions and insertions. The Illumina Platinum Genomes along with Real Time Genomics have developed high-confidence phased calls for this genome by using the extensive (17-member) pedigree to which this sample belongs and phasing variants through hereditary tracks.

NIST released the pilot RM 8398 in May 2015 [8]. Then, 8,300 vials of this RM were available for purchase, each containing approximately 10 µg of DNA extracted from a large homogeneous culture produced by the Coriell Repositories. Homogeneity of the DNA aliquots was verified by sequencing six randomly selected vials of the RM to approximately 300X. No differences in SNP, indels, or copy number variants were detected among DNA samples sequenced. Stability was assessed by measuring DNA fragment size distributions with pulsed field gel electrophoresis under a variety of conditions, including storage at elevated temperatures, as a proxy for accelerating aging.

In addition to the pilot RM 8398, GIAB has selected an AJ trio (i.e., samples from a child and both parents) and an Asian trio from the Personal Genome Project for characterization [35]. These genomes are also available from the Coriell Repositories as Epstein-Barr virus (EBV)-immortalized B lymphoblastoid cell lines and extracted DNA. Four of these genomes will be available as components of NIST RMs, with a planned release in mid-2016. The genomic DNA from the son of the AJ trio will be distributed as RM 8391, the DNA from the AJ trio (parents and son) will be distributed as RM 8392, and the DNA corresponding to the son of the Asian trio will be distributed as RM 8393 (note that the genomes of the Asian trio parents are only available from Coriell). Similar to the pilot genome, these NIST RMs consist of DNA extracted from a large batch of cells. These genomes were consented under the compressive Personal Genome Project [35] consent, which permits broad application of the genomes as scientific resources. For example, commercial redistribution of these samples is permitted, and two companies have already produced products that contain materials derived from biological samples of the NIST AJ son from whom RM DNA is prepared.

GIAB has established a collection of publically available data for these genomes from a comprehensive suite of technologies and methods. GIAB has formed an Analysis Group to coordinate analyses and integration of these diverse data types. The Analysis Group has the goal of forming high-confidence phased variant calls of all types and sizes for as much of the genome as possible.

These whole genome RMs are characterized for more than 2 billion bp. Therefore, instead of testing a laboratory's accuracy for a single pathogenic variant, these RMs can be used to establish test accuracy for different types of variants in different genome contexts. Standardizing definitions and computation of genome-scale performance metrics is complex, so GIAB and the Global Alliance for Genomics and Health formed a Benchmarking Team [36]. This team is developing standardized definitions for performance metrics along with open-source tools to calculate these metrics, even in difficult cases where different tools produce different representations of complex variants (i.e., from nearby SNPs and/or indels). In addition, the team is developing methods to stratify performance by variant type, genome context, functional context, and data characteristics. For example, they have compiled standardized bed files with different types of repetitive regions, GC content, etc. for stratification by genomic context. These files can facilitate the understanding of genome regions with relatively high or low accuracy, as well as regions in which the RMs have few or no examples of mutations. The performance metrics definitions, variant comparison tools, and stratification methods developed by the Benchmarking Team will help laboratories characterize the performance of variant calls in an accurate and standardized way.

GIAB is considering developing other types of RMs, including synthetic spike-in DNA that could be added into any sample to test the mutation detection accuracy. These spike-ins could be particularly useful for detecting somatic mutations at different allele frequencies in tumor samples. For germline sequencing, these spike-ins might include mutations that are difficult to detect (e.g., truncating indels and complex variants) in exons of genes that are typically included in targeted gene panels. Characterizing an assay's capability of identifying difficult mutations in disease-related genes is important because many disease-related mutations are difficult to detect, and few or no examples of these mutations exist in disease-related genes in the "normal" genomes that GIAB is developing as RMs.

NGS assays are becoming common in clinical laboratories. Use of this technology allows the rapid, economical analysis of multi-gene panels, exomes, or genomes for disease-causing variants. Before the introduction of NGS technology, clinical laboratories analyzed single genes or small gene panels using Sanger sequencing. Validating the Sanger assays that covered small, well-studied genomic regions could be achieved by careful characterization of a relatively small number of residual patient samples, or characterized genomic DNA derived from cell lines containing variants in the gene(s) of interest. Through analysis of these samples, laboratories could determine their capability to detect the expected variants, measure the number of false positive or negative results, and study the repeatability and reproducibility of the assay, as required by regulatory requirements and recommended by professional guidelines [1617181920213738]. It is much more difficult to perform analytical validation of large NGS gene panels and exome or genome tests than it is to validate simpler tests. It is not possible to obtain or analyze genomic DNA samples that contain pathogenic variants in every gene of the human genome. It is also extremely difficult to obtain samples without variants that are sufficiently well-characterized for regions included in assays to allow for the calculation of false positive and negative rates. The CDC Next-generation Sequencing: Standardization of Clinical Testing [Nex-StoCT] workgroup [4], the American College of Medical Genetics [39], and the College of American Pathologists (CAP) [18] recommend that laboratories use well-characterized genomic DNA RMs with both naturally occurring and disease-associated variants. The genomic DNA RMs created by NIST helps to fill this need.

Laboratories that use NIST RMs need tools to assess the capability of their methodology to detect variants and accurately sequence non-variant genomic regions. The GeT-RM program has worked with the National Center for Biotechnology Information (NCBI), in collaboration with members of the genetic testing community, academic and clinical genetic testing laboratories, in vitro diagnostics (IVD) manufacturers, professional organizations, other government agencies, and the Coriell Cell Repositories to create an interactive web portal called the GeT-RM Browser [40]. This tool allows for comparison of the NA12878 sequence generated by laboratories either from the NIST reference material, or from genomic DNA purchased from Coriell, to the high confidence variant calls that are associated with the NIST RM 8398 [15]. The GeT-RM Browser also contains a number of high quality, publicly available datasets, including data from the Illumina Platinum Genome [41], 36 datasets from NA12878, and 21 datasets for NA19240 generated by eight clinical laboratories and four research laboratories as part of a GeT-RM project. The browser allows easy access to and visualization of public variant call sets and the alignments used to generate them. The GeT-RM Browser also provides a number of other data tracks that show features such as segmental duplications, reference SNPs, and genomic regions that are difficult to sequence, that laboratories can also use to assess the quality of their data. Data from the browser can be downloaded and compared to user data, or the user can upload NA12878 or NA19240 sequence data and use the browser to visually compare their calls to those in the available data sets. New datasets for NA12878, as well as datasets and high-confidence calls from the candidate NIST RMs from the Personal Genome Project, will be added to the browser as they become available.