The aim of prediction modeling is to develop an accurate and useful clinical prediction model with multiple variables using comprehensive datasets. First, we have to articulate several important research questions that affect database selection and the approach of model generation. (1) What is the target outcome (event or disease) to predict (e.g., diabetes, CVD, or fracture)? (2) Who is the target patient of the model (e.g., general population, elderly population ≥65 years or patients with type 2 diabetes)? (3) Who is the target user of the prediction model (e.g., layperson, doctor or health-related organization)? Depending on the answers to the above questions, researchers can choose the proper datasets for the model. The category of target users will determine the selection and handling process of multiple variables, which will affect the structure of the clinical prediction model. For example, if researchers want to make a prediction model for laypersons, a simple model with not many user-friendly questions in only a few categories (e.g., yes vs. no) could be ideal.

The dataset is one of the most important components of the clinical prediction model—often not under investigators' control—and ultimately determines its quality and credibility; however, there are no general rules for assessing the quality of data [9]. Yet, there is no such thing as perfect data and prefect model. It would be reasonable to search for best-suited dataset. Oftentimes, secondary or administrative data sources must be utilized because a primary dataset with the study endpoint and all of key predictors is not available. Researchers should use different types of datasets, depending on the purpose of the prediction model. For example, a model for screening high-risk individuals with undiagnosed condition/disease can be developed using cross-sectional cohort data. However, such models may have relatively low power for predicting future incidence of disease when different risk factors come into play. Accordingly, longitudinal or prospective cohort datasets should be used for prediction models for future events (Table 1). Models for prevalent events are useful for predicting asymptomatic diseases, such as diabetes or chronic kidney disease, by screening undiagnosed cases, whereas models for incident events are useful for predicting the incidence of relatively severe diseases, such as CVD, stroke, and cancer.

A universal clinical prediction model for disease does not exist; thus, separate specific models that can individually assess the role of ethnicity, nationality, sex, or age on disease risk are warranted. For example, the Framingham coronary heart disease (CHD) risk score is generated by one of the most commonly used clinical prediction models; however, it tends to overestimate CHD risk by approximately 5-fold in Asian populations [1718]. This indicates that models derived from one ethnicity sample may not be directly applied to populations of other ethnicities. Other specific characteristics of study populations beside ethnicity (e.g., obesity- or culture-related variables) could be important.

There is no absolute consensus on the minimal requirement for dataset sample size. Generally, large representative, contemporary datasets that closely reflect the characteristics of their target population are ideal for modeling and can enhance the relevance, reproducibility, and generalizability of the model. Moreover, two types of datasets are generally needed: a development dataset and a validation dataset. A clinical prediction model is first derived from analyses of the development dataset and its predictive performance should be assessed in different populations based on the validation dataset. It is highly recommended to use validation datasets from external study populations or cohorts, whenever available [1920]; however, if it is not possible to find appropriate external datasets, an internal validation dataset can be formed by randomly splitting the original cohort into two datasets (if sample size is large) or statistical techniques such as jackknife or bootstrap resampling (if not) [21]. The splitting ratio can vary depending on the researchers' particular goals, but generally, more subjects should be allocated to the development dataset than to the validation dataset.

Since cohort datasets contain more variables than can reasonably be used in a prediction model, evaluation and selection of the most predictive and sensible predictors should be done. Generally, inclusion of more than 10 variables/questions may decrease the efficiency, feasibility and convenience of prediction models, but expert's judgment that could be somewhat subjective is required to assess the need for each situation. Predictors that were previously found to be significant should normally be considered as candidate variables (e.g., family history of diabetes in diabetes risk score). It should be noted that not all significant predictors need to be included in the final model (e.g., P<0.05); predictor selection must be always guided by clinical relevance/judgement to prevent nonsensical or less relevant or user-unfriendly variables (e.g., socioeconomic status-related) or possible false-positive associations. Additionally, variables which are highly correlated with others may be excluded because they contribute little unique information [22]. On the other hand, variables not statistically significant or with small effect size may still contribute to the model [23]. Depending on researcher discretion, different models that analyze different variables may be developed for targeting distinct users. For example, a simple clinical prediction model that does not require laboratory variables and a comprehensive model that does could both be designed for laypersons and health care providers, respectively [19].

With regard to variable coding, categorical and continuous variables should be managed differently [8]. For ordered categorical variables, infrequent categories can be merged and similar variables may be combined/grouped. For example, past and current smoker categories can be merged if numbers of subjects who report being a past or current smoker are relatively small and variable unification does not alter the statistical significance of the model materially. Although continuous parameters are usually included in a regression model, assuming linearity, researchers should consider the possibility of non-linear associations such as J- or U-shaped distributions [24]. Furthermore, the relative effect of a continuous variable is determined by the measurement scale used in the model [8]. For example, the impact of fasting glucose levels on the risk of CVD may be interpreted as having a stronger influence when scaled per 10 mg/dL than per 1 mg/dL.

Researchers often emphasize the importance of not dichotomizing continuous variables in the initial stage of model development because valuable predictive information can be lost during categorization [24]. However, prediction models—is not the same thing as regression models—with continuous parameters may be complex and hard to use or be understood by laypersons, because they have to calculate their risk scores by themselves. A web or computer-based platform is usually required for the implementation of these models. Otherwise, in a later phase, researchers may transform the model into a user-friendly format by categorizing some predictors, if the predictive capacity of the model is retained [81925].

Finally, missing data is a chronic problem in most data analyses. Missing data can occur various reasons, including uncollected (e.g., by design), not available or not applicable, refusal by respondent, dropout, or "don't know." To handle this issue, researchers may consider imputation technique, dichotomizing the answer into yes versus others, or allow "unknown" as a separate category as in http://www.cancer.gov/bcrisktool/.

Although there are no consensus guidelines for choosing variables and determining structures to develop the final prediction model, various strategies with statistical tools are available [89]. Regression analyses, including linear, logistic, and Cox models are widely used depending on the model and its intended purpose. First, the full model approach is to include all the candidate variables in the model; the benefit of this approach is to avoid overfitting and selection bias [9]. However, it can be impractical to pre-specify all predictors and previously significant predictors may not be in a new population/sample. Second, a backward elimination approach or stepwise selection method can be applied to remove a number of insignificant candidate variables. To check for overfitting of the model, Akaike information criterion (AIC) [26], an index of model fitting that charges a penalty against larger models, may be useful [19]. Lower AIC values indicate a better model fit. Some interpret that AIC addresses explanation and Bayesian information criterion (BIC) addresses prediction, where BIC may be considered a Bayesian counterpart [27].

If researchers prefer algorithm modeling culture instead of data modeling culture, e.g., formula-based regression [28], a classification and regression tree analysis or recursive partitioning could be considered [282930].

With regard to determining scores for each predictor in the generation of simplified models, researchers using expert judgment may create a weighted scoring system by converting β coefficients [19] or odds ratios [20] from the final model to integer values, while preserving monotonicity and simplicity. For example, from the logistic regression model built by Lee et al. [19], β coefficients <0.6, 0.7 to 1.3, 1.4 to 2.0, and >2.1 were assigned scores of 1, 2, 3, and 4, respectively.

After model generation, researchers should evaluate the predictive power of their proposed model using an independent dataset, where truly external dataset is preferred whenever available. There are several standard performance measures that capture different aspects: two key components are calibration and discrimination [8931]. Calibration can be assessed by plotting the observed proportions of events against the predicted probabilities for groups defined by ranges of individual predicted risk [910]. For example, a common method is to categorize 10 risk groups of equal size (deciles) and then conduct the calibration process [32]. The most ideal calibration plot would show a 45° line, which indicates that the observed proportions of events and predicted probabilities completely overlap over the entire range of probabilities [9]. However, this is not guaranteed when external validation is conducted with a different sample. Discrimination is defined as the ability to distinguish events versus non-events (e.g., dead vs. alive) [8]. The most common discrimination measure is the AUC or, equivalently, concordance (c)-statistic. The AUC is equal to the probability that, given two individuals randomly selected—one who will develop an event and another who will not—the model will assign a higher probability of an event to the former [10]. A c-statistic value of 0.5 indicates a random chance (i.e., flip of a coin). The usual c-statistic range for a prediction model is 0.6 to 0.85; this range can be affected by target-event characteristics (disease) or the study population. A model with a c-statistic ranging from 0.70 to 0.80 has an adequate power of discrimination; a range of 0.80 to 0.90 is considered excellent. Table 2 shows several common statistical measures for model evaluation.

As usual, selection, application and interpretation of any statistical method and results need great care as virtually all methods entail assumptions and limited capacity. Let us review some here. Predictive values depend on the disease prevalence so direct comparison for different diseases may not be valid. When sample size is very large, P value can be impressively small even for a practically meaningless difference. Net reclassification index and integrated discrimination improvement are known to lead to non-proper scoring and vulnerable to miscalibrated or overfit problems [33]. AUC and R2 are often hard to increase by a new predictor, even with large odds ratio. Despite similar names, AIC and BIC address slightly different issues and information in BIC can be decreased with sample size increases. The Hosmer-Lemeshow test is highly sensitive when sample size is large, which is not an ideal property as a goodness-fit statistic. Calibration plot can easily provide a high correlation coefficient (>0.9), simply because they are computed for predicted versus observed values on grouped data (without random variability). Finally, AUC also needs caution: a high value (e.g., >0.9) may mean excellent discrimination but it can also reflect the situation where prediction is not so relevant: (1) the task is closer to diagnostic or early onset rather than prediction; (2) cases vs. non-cases are fundamentally different with minimal overlap; or (3) predictors and endpoints are virtually the same things (e.g., current blood pressure vs. future blood pressure).

Despite the long list provided above, we do not think this is a discouraging news to researchers. We may tell us no method is perfect and "one size does not fit all" is also true to statistical methods; thus blinded or automated application can be dangerous.

It is crucial to separate internal and external validation and to conduct the previously mentioned analyses on both datasets to finalize the research findings (see the following for example reports [192034]). Internal validation can be done using a random subsample or different years from the development dataset or by conducting bootstrap resampling [22]. This approach can particularly assess the stability of selected predictors, as well as prediction quality. Subsequently, external validation should be performed on an independent dataset from that which was previously used to develop the model. For example, datasets can be obtained from populations from other hospitals or centers (see geographic validation [19]) or a more recently collected cohort population (temporal validation [34]). This process is often considered to be a more powerful test for prediction models than internal validation because it evaluates transportability, generalizability and true replication, rather than reproducibility [8]. Poor model performance may occur after use of an external dataset due to differences in healthcare systems, measurement methods/definitions of predictors and/or endpoint, subject characteristics or context (e.g., high vs. low risk).