Statistical notes for clinical researchers: logistic regression

Hae-Young Kim

doi:10.5395/rde.2017.42.4.342

Logistic regression is a regression model where the dependent variable is categorical and corresponding independent variables can be categorical or continuous. This article covers the case of a binary dependent variable such as an event occurring coded 1 = ‘event’ and 0 = ‘no event’. Frequent outcomes are pass/fail, win/lose, disease/no disease, etc. The logistic regression model estimates the probability that an event occurs versus the probability that the event does not occur.

An example: score and pass data

Let's say that an institution performed an assessment procedure to determine pass and fail of the participants considering exam scores, interview result, and reputation among colleagues. Table 1 shows a data with 2 variables, exam scores and pass state (1 = pass, 0 = fail). We can notice that there is a trend that persons with lower scores are more likely to fail, while persons with higher scores tend to pass. When we plot the data as Figure 1A, we can see persons with value 1 (pass) have scores that shift to the right side, while persons with value 0 (fail) have those that shift to the left side. Persons with same score may not have the same outcome (e.g., cases of score = 799) because the assessment procedure comprises other factors. At least we can postulate that the probability of pass may be higher if the score is higher. What is the best-fit line for this data? A usual straight regression line ranging from minus infinity to infinity does not make sense for this case. Instead of ordinal regression the logistic regression can fit the probability more adequately. In Figure 1B, the probability estimated by logistic regression is presented. The estimated probability by the logistic regression model (red dot and line) seems reasonable because it reflects the observed reality that the probability of pass decreases close to zero with very low scores, while the probability increases close to one with very high scores.

Table 1

Scores of applicants who passed the final assessments

Score	755	755	763	781	783	788	792	793	798	799	799	802	813	824	845
Pass	0	0	0	0	1	1	0	1	0	0	1	1	1	1	1

Figure 1

Scatterplot of pass (1 = pass, 0 = fail) and score: (A) pass and score, and (B) estimated probability (P) of pass added.

Review of probability, odds, and odds ratio

From the previous sections about risk, odds, and odds ratio, they were defined as following formulas:

P r o b a b i l i t y o r r i s k (p) = \frac{n u m b e r o f e v e n t s}{n u m b e r o f a l l o b s e r v a t i o n s}

O d d s = \frac{p (e v e n t)}{p (n o n - e v e n t)} = \frac{p}{1 - p}

O d d s r a t i o = \frac{{o d d s}_{1}}{{o d d s}_{0}} = \frac{\frac{p_{1}}{1 - p_{1}}}{\frac{p_{0}}{1 - p_{0}}}

Let's consider an example of flipping of fair coins vs. loaded coins.

Odds ratio is important in interpreting in logistic regression because it represents how much the odds change with 1 unit increase in the predictor variables while keeping all other variables constant.

Logistic regression

1. Logit link function

Logistic regression uses logit link function to estimate unknown probability of outcome (p) for a linear combination of predictor variables. The original probability ranging from zero to one cannot match with linear combination of predictor variables ranging minus infinity to infinity [1].

L o g i t (p) = l n (o d d s) = {l o g}_{e} (\frac{p}{1 - p})

where log_ex = ln (x) and e = Euler's number, 2.71828.

Logit link function accommodate p ranging from zero to one. The logit link function reconciles the incongruity by changing the range of dependent variable, p, into minus infinity to infinity. As seen in Table 2, final logit (p) values cover from minus values to plus values.

Table 2

Logit transformation from probability (p)

P	0.01	0.1	0.2	0.3	0.4	0.5	0.6	0.7	0.8	0.9	0.99
1−p	0.99	0.9	0.8	0.7	0.6	0.5	0.4	0.3	0.2	0.1	0.01
odds	0.01	0.11	0.25	0.43	0.67	1.00	1.50	2.33	4.00	9.00	99.00
Logit (p) = ln (odds)	−4.60	−2.20	−1.39	−0.85	−0.41	0.00	0.41	0.85	1.39	2.20	4.60

2. Property of logit and inverse logit

Shown in Figure 2A, logit function has an s-shaped curve. Logit (p) is undefined at p = 0 and p = 1. When p approaches close to zero, the value of logit (p) goes toward minus infinity and when p get larger close to one, it goes toward infinity. We can notice that the logit (p) has a value of zero at p = 0.5.

Figure 2B shows inverse logit graph. Inverse logit returns the probability of the event ranging from zero to one. Figure 1B and Figure 2B show similar shape because both represent estimated probability. The induced inverse logit formula is as following:

I n v e r s e l o g i t (p) = {l o g}^{- 1} (\frac{p}{1 - p}) = \frac{1}{1 + e^{- α}} = \frac{e^{α}}{1 + e^{α}} = p

where α = some number.

Figure 2

Probability and logit transformation: (A) natural log of adds ratio (logit [p]), (B) inverse logit (p).

3. Estimation of logistic regression equation

Simple logistic regression is expressed as logit (p) and linear combination of predictor variables as below.

l o g i t (p) = \ln (\frac{p}{1 - p}) = β_{0} + β_{1} x

Using a fictitious data based on the example above logistic regression was performed and the output was provided (pages 6–7). The observations (n = 15) are multiplied by 100 to provide high power to get significant estimates artificially. The dependent variable was the binary variable pass and score was the predictor variable. The SPSS (IBM Corp., Armonk, NY, USA) output of (e) below gives coefficients as following.

The estimated logistic equation is:

l o g i t (p) = \ln (\frac{p}{1 - p}) = β_{0} + β_{1} x = - 73.578140 + 0.093115 \times (S c o r e)

where p = probability of ‘pass’.

Here represents odds ratio which means the amount of change in odds with 1 unit increase in the predictor variable. The odds ratio, exp (β₁) = e^0.093115 = 1.097588. Therefore, as the score increases by 1 point, the odds of pass was estimated to increase by 9.8%. The 95% confidence interval of odds ratio was [1.086, 1.109] which does not include a value one. Odds ratio value of one means that 1 unit increase in the predictor variable does not make any difference in odds. Therefore, to get statistical significance, it is important to confirm that 95% confidence interval of odds ratio does not include one.

1) Estimated probability

After some algebra, inverse logit gives us the estimated probability by the predictor variable as follows:

l o g i t (p) = \ln (\frac{p}{1 - p}) = β_{0} + β_{1} x

\frac{p}{1 - p} = e^{β_{0} + β_{1} x} \to p = e^{β_{0} + β_{1} x} (1 - p) \to p (1 + e^{β_{0} + β_{1} x}) = e^{β_{0} + β_{1} x} \to \hat{p} = \frac{e^{β_{0} + β_{1} x}}{1 + e^{β_{0} + β_{1} x}}

To get the probability of pass at score 781, we can use the estimated probability function. Also, if the score increases by one point to 782 then the estimated probability can be calculated as shown in Table 3. According to the results for the score 781, estimated probability of pass in the assessment is 0.30 or 30%. Also, the odds ratio is obtained as 1.098, which is the same value with exp (β₁) from the SPSS output, representing the increase of odds of 9.8% related to a 1 point increase of the score.

Table 3

Estimated probability and odds ratio based on logistic regression model

Score = 781	$\hat{p} = \frac{e^{β_{0} + β_{1} x}}{1 + e^{β_{0} + β_{1} x}} = \frac{e^{- 73.578140 + 0.093115 (781)}}{1 + e^{- 73.578140 + 0.093115 (781)}} = \frac{e^{- 0.85533}}{1 + e^{- 0.85533}} = \frac{0.425145}{1 + 0.425145} = 0.298317$
	$odds = \frac{p}{1 - p} = \frac{0.298317}{1 - 0.298317} = 0.425145$
Score = 782	$\hat{p} = \frac{e^{β_{0} + β_{1} x}}{1 + e^{β_{0} + β_{1} x}} = \frac{e^{- 73.578140 + 0.093115 (782)}}{1 + e^{- 73.578140 + 0.093115 (782)}} = \frac{e^{- 0.76221}}{1 + e^{- 0.76221}} = \frac{0.466634}{1 + 0.466634} = 0.318167$
	$odds = \frac{p}{1 - p} = \frac{0.318167}{1 - 0.318167} = 0.466634$
Odds ratio for a 1 point increase in score: $\frac{odds at 782}{odds at 781} = \frac{0.466634}{0.425145} = 1.097588$

Estimated probability for other score values are shown in the SPSS output (f) below under ‘PRE_1’. Using this we can calculate odds and odds ratio between 2 specific scores. For example, suppose my present score is 781 and I'd like to know how much increase in odds if I raise my score by 11 points and get 792. Then the odds ratio can be obtained easily. The calculation ends up to an increase of 179% in odds when I raise up my score by 11 points (Table 4).

\frac{o d d s a t 792}{o d d s a t 781} = \frac{1.18}{0.43} = 2.79

Table 4

Scores, estimated probabilities, and odds ratios based on logistic regression model

Score	755	755	763	781	783	788	792	793	798	799	799	802	813	824	845
P	0.04	0.04	0.07	0.30	0.34	0.45	0.54	0.57	0.67	0.69	0.69	0.75	0.89	0.96	0.99
Odds	0.04	0.04	0.08	0.43	0.51	0.82	1.18	1.30	2.07	2.27	2.27	3.01	8.37	23.31	164.84