Student's t-test

The independent samples t-test is used when two separate sets of independent and identically distributed samples are obtained, and one variable from each of the two populations is compared. For example, suppose we are evaluating the effect of a medical treatment, and we enroll 100 subjects into our study, then randomly assign 50 subjects to the treatment group and 50 subjects to the control group. In this case, we have two independent samples and would use the unpaired form of the t-test.

Paired samples t-tests typically consist of a sample of matched pairs of similar units, or one group of units that has been tested twice (a "repeated measures" t-test).

A typical example of the repeated measures t-test would be where subjects are tested prior to a treatment, say for high blood pressure, and the same subjects are tested again after treatment with a blood-pressure-lowering medication. By comparing the same patient's numbers before and after treatment, we are effectively using each patient as their own control. That way the correct rejection of the null hypothesis (here: of no difference made by the treatment) can become much more likely, with statistical power increasing simply because the random interpatient variation has now been eliminated. However, an increase of statistical power comes at a price: more tests are required, each subject having to be tested twice. Because half of the sample now depends on the other half, the paired version of Student's t-test has only n/2 − 1 degrees of freedom (with n being the total number of observations). Pairs become individual test units, and the sample has to be doubled to achieve the same number of degrees of freedom. Normally, there are n − 1 degrees of freedom (with n being the total number of observations).[22]

A paired samples t-test based on a "matched-pairs sample" results from an unpaired sample that is subsequently used to form a paired sample, by using additional variables that were measured along with the variable of interest.[23] The matching is carried out by identifying pairs of values consisting of one observation from each of the two samples, where the pair is similar in terms of other measured variables. This approach is sometimes used in observational studies to reduce or eliminate the effects of confounding factors.

Paired samples t-tests are often referred to as "dependent samples t-tests".

In testing the null hypothesis that the population mean is equal to a specified value μ₀, one uses the statistic

${\displaystyle t={\frac {{\bar {x}}-\mu _{0}}{s/{\sqrt {n}}}},}$

where ${\displaystyle {\bar {x}}}$ is the sample mean, s is the sample standard deviation and n is the sample size. The degrees of freedom used in this test are n − 1. Although the parent population does not need to be normally distributed, the distribution of the population of sample means ${\displaystyle {\bar {x}}}$ is assumed to be normal.

By the central limit theorem, if the observations are independent and the second moment exists, then ${\displaystyle t}$ will be approximately normal ${\displaystyle N(0;1)}$.

Suppose one is fitting the model

${\displaystyle Y=\alpha +\beta x+\varepsilon ,}$

where x is known, α and β are unknown, ε is a normally distributed random variable with mean 0 and unknown variance σ², and Y is the outcome of interest. We want to test the null hypothesis that the slope β is equal to some specified value β₀ (often taken to be 0, in which case the null hypothesis is that x and y are uncorrelated).

Let

${\displaystyle {\begin{aligned}{\hat {\alpha }},{\hat {\beta }}&={\text{least-squares estimators}},\\SE_{\hat {\alpha }},SE_{\hat {\beta }}&={\text{the standard errors of least-squares estimators}}.\end{aligned}}}$

Then

${\displaystyle t_{\text{score}}={\frac {{\hat {\beta }}-\beta _{0}}{SE_{\hat {\beta }}}}\sim {\mathcal {T}}_{n-2}}$

has a t-distribution with n − 2 degrees of freedom if the null hypothesis is true. The standard error of the slope coefficient:

${\displaystyle SE_{\hat {\beta }}={\frac {\sqrt {\displaystyle {\frac {1}{n-2}}\sum _{i=1}^{n}(y_{i}-{\hat {y}}_{i})^{2}}}{\sqrt {\displaystyle \sum _{i=1}^{n}(x_{i}-{\bar {x}})^{2}}}}}$

can be written in terms of the residuals. Let

${\displaystyle {\begin{aligned}{\hat {\varepsilon }}_{i}&=y_{i}-{\hat {y}}_{i}=y_{i}-({\hat {\alpha }}+{\hat {\beta }}x_{i})={\text{residuals}}={\text{estimated errors}},\\{\text{SSR}}&=\sum _{i=1}^{n}{{\hat {\varepsilon }}_{i}}^{2}={\text{sum of squares of residuals}}.\end{aligned}}}$

Then t_score is given by

${\displaystyle t_{\text{score}}={\frac {({\hat {\beta }}-\beta _{0}){\sqrt {n-2}}}{\sqrt {\frac {SSR}{\sum _{i=1}^{n}(x_{i}-{\bar {x}})^{2}}}}}.}$

Another way to determine the t_score is

${\displaystyle t_{\text{score}}={\frac {r{\sqrt {n-2}}}{\sqrt {1-r^{2}}}},}$

where r is the Pearson correlation coefficient.

The t_{score, intercept} can be determined from the t_{score, slope}:

${\displaystyle t_{\text{score,intercept}}={\frac {\alpha }{\beta }}{\frac {t_{\text{score,slope}}}{\sqrt {s_{\text{x}}^{2}+{\bar {x}}^{2}}}},}$

where s_x² is the sample variance.

Given two groups (1, 2), this test is only applicable when:

• the two sample sizes are equal,
• it can be assumed that the two distributions have the same variance.

Violations of these assumptions are discussed below.

The t statistic to test whether the means are different can be calculated as follows:

${\displaystyle t={\frac {{\bar {X}}_{1}-{\bar {X}}_{2}}{s_{p}{\sqrt {\frac {2}{n}}}}},}$

where

${\displaystyle s_{p}={\sqrt {\frac {s_{X_{1}}^{2}+s_{X_{2}}^{2}}{2}}}.}$

Here s_p is the pooled standard deviation for n = n₁ = n₂, and s ²
_X1 and s ²
_X2 are the unbiased estimators of the population variance. The denominator of t is the standard error of the difference between two means.

For significance testing, the degrees of freedom for this test is 2n − 2, where n is sample size.

This test is used only when it can be assumed that the two distributions have the same variance (when this assumption is violated, see below). The previous formulae are a special case of the formulae below, one recovers them when both samples are equal in size: n = n₁ = n₂.

The t statistic to test whether the means are different can be calculated as follows:

${\displaystyle t={\frac {{\bar {X}}_{1}-{\bar {X}}_{2}}{s_{p}\cdot {\sqrt {{\frac {1}{n_{1}}}+{\frac {1}{n_{2}}}}}}},}$

where

${\displaystyle s_{p}={\sqrt {\frac {(n_{1}-1)s_{X_{1}}^{2}+(n_{2}-1)s_{X_{2}}^{2}}{n_{1}+n_{2}-2}}}}$

is the pooled standard deviation of the two samples: it is defined in this way so that its square is an unbiased estimator of the common variance, whether or not the population means are the same. In these formulae, n_i − 1 is the number of degrees of freedom for each group, and the total sample size minus two (that is, n₁ + n₂ − 2) is the total number of degrees of freedom, which is used in significance testing.

This test, also known as Welch's t-test, is used only when the two population variances are not assumed to be equal (the two sample sizes may or may not be equal) and hence must be estimated separately. The t statistic to test whether the population means are different is calculated as

${\displaystyle t={\frac {{\bar {X}}_{1}-{\bar {X}}_{2}}{s_{\bar {\Delta }}}},}$

where

${\displaystyle s_{\bar {\Delta }}={\sqrt {{\frac {s_{1}^{2}}{n_{1}}}+{\frac {s_{2}^{2}}{n_{2}}}}}.}$

Here s_i² is the unbiased estimator of the variance of each of the two samples with n_i = number of participants in group i (i = 1 or 2). In this case ${\displaystyle (s_{\bar {\Delta }})^{2}}$ is not a pooled variance. For use in significance testing, the distribution of the test statistic is approximated as an ordinary Student's t-distribution with the degrees of freedom calculated using

${\displaystyle {\text{d.f.}}={\frac {\left({\frac {s_{1}^{2}}{n_{1}}}+{\frac {s_{2}^{2}}{n_{2}}}\right)^{2}}{{\frac {(s_{1}^{2}/n_{1})^{2}}{n_{1}-1}}+{\frac {(s_{2}^{2}/n_{2})^{2}}{n_{2}-1}}}}.}$

This is known as the Welch–Satterthwaite equation. The true distribution of the test statistic actually depends (slightly) on the two unknown population variances (see Behrens–Fisher problem).

The test[24] deals with the famous Behrens–Fisher problem, i.e., comparing the difference between the means of two normally distributed populations when the variances of the two populations are not assumed to be equal, based on two independent samples.

The test is developed as an exact test that allows for unequal sample sizes and unequal variances of two populations. The exact property still holds even with small extremely small and unbalanced sample sizes (e.g. ${\displaystyle n_{1}=5,n_{2}=50}$).

The statistic to test whether the means are different can be calculated as follows:

Let ${\displaystyle X=[X_{1},X_{2},\ldots ,X_{m}]^{T}}$ and ${\displaystyle Y=[Y_{1},Y_{2},\ldots ,Y_{n}]^{T}}$ be the i.i.d. sample vectors (${\displaystyle m>n}$) from ${\displaystyle N(\mu _{1},\sigma _{1}^{2})}$ and ${\displaystyle N(\mu _{2},\sigma _{2}^{2})}$ separately.

Let ${\displaystyle (P^{T})_{n\times n}}$ be an ${\displaystyle n\times n}$ orthogonal matrix whose elements of the first row are all ${\displaystyle 1/{\sqrt {n}}}$, similarly, let ${\displaystyle (Q^{T})_{n\times m}}$ be the first n rows of an ${\displaystyle m\times m}$ orthogonal matrix (whose elements of the first row are all ${\displaystyle 1/{\sqrt {m}}}$).

Then ${\displaystyle Z:=(Q^{T})_{n\times m}X/{\sqrt {m}}-(P^{T})_{n\times n}Y/{\sqrt {n}}}$ is an n-dimensional normal random vector.

${\displaystyle Z\sim N((\mu _{1}-\mu _{2},0,...,0)^{T},({\frac {\sigma _{1}^{2}}{m}}+{\frac {\sigma _{2}^{2}}{n}})I_{n}).}$

From the above distribution we see that

${\displaystyle Z_{1}-(\mu _{1}-\mu _{2})\sim N(0,{\frac {\sigma _{1}^{2}}{m}}+{\frac {\sigma _{2}^{2}}{n}}),}$

${\displaystyle {\frac {\sum _{i=2}^{n}Z_{i}^{2}}{n-1}}\sim {\frac {\chi _{n-1}^{2}}{n-1}}\times ({\frac {\sigma _{1}^{2}}{m}}+{\frac {\sigma _{2}^{2}}{n}})}$

${\displaystyle Z_{1}-(\mu _{1}-\mu _{2})\perp \sum _{i=2}^{n}Z_{i}^{2}.}$

${\displaystyle T_{e}:={\frac {Z_{1}-(\mu _{1}-\mu _{2})}{\sqrt {(\sum _{i=2}^{n}Z_{i}^{2})/(n-1)}}}\sim t_{n-1}.}$

This test is used when the samples are dependent; that is, when there is only one sample that has been tested twice (repeated measures) or when there are two samples that have been matched or "paired". This is an example of a paired difference test. The t statistic is calculated as

${\displaystyle t={\frac {{\bar {X}}_{D}-\mu _{0}}{s_{D}/{\sqrt {n}}}},}$

where ${\displaystyle {\bar {X}}_{D}}$ and ${\displaystyle s_{D}}$ are the average and standard deviation of the differences between all pairs. The pairs are e.g. either one person's pre-test and post-test scores or between-pairs of persons matched into meaningful groups (for instance, drawn from the same family or age group: see table). The constant μ₀ is zero if we want to test whether the average of the difference is significantly different. The degree of freedom used is n − 1, where n represents the number of pairs.

Example of matched pairs Pair Name Age Test 1 John 35 250 1 Jane 36 340 2 Jimmy 22 460 2 Jessy 21 200

Example of repeated measures Number Name Test 1 Test 2 1 Mike 35% 67% 2 Melanie 50% 46% 3 Melissa 90% 86% 4 Mitchell 78% 91%

If the approach for unequal variances (discussed above) is followed, the results are

${\displaystyle {\sqrt {{\frac {s_{1}^{2}}{n_{1}}}+{\frac {s_{2}^{2}}{n_{2}}}}}\approx 0.04849}$

and the degrees of freedom

${\displaystyle {\text{d.f.}}\approx 7.031.}$

The test statistic is approximately 1.959, which gives a two-tailed test p-value of 0.09077.

If the approach for equal variances (discussed above) is followed, the results are

${\displaystyle s_{p}\approx 0.08396}$

and the degrees of freedom

${\displaystyle {\text{d.f.}}=10.}$

The test statistic is approximately equal to 1.959, which gives a two-tailed p-value of 0.07857.

The t-test provides an exact test for the equality of the means of two i.i.d. normal populations with unknown, but equal, variances. (Welch's t-test is a nearly exact test for the case where the data are normal but the variances may differ.) For moderately large samples and a one tailed test, the t-test is relatively robust to moderate violations of the normality assumption.[25] In large enough samples, the t-test asymptotically approaches the z-test, and becomes robust even to large deviations from normality.[17]

If the data are substantially non-normal and the sample size is small, the t-test can give misleading results. See Location test for Gaussian scale mixture distributions for some theory related to one particular family of non-normal distributions.

When the normality assumption does not hold, a non-parametric alternative to the t-test may have better statistical power. However, when data are non-normal with differing variances between groups, a t-test may have better type-1 error control than some non-parametric alternatives.[26] Furthermore, non-parametric methods, such as the Mann-Whitney U test discussed below, typically do not test for a difference of means, so should be used carefully if a difference of means is of primary scientific interest.[17] For example, Mann-Whitney U test will keep the type 1 error at the desired level alpha if both groups have the same distribution. It will also have power in detecting an alternative by which group B has the same distribution as A but after some shift by a constant (in which case there would indeed be a difference in the means of the two groups). However, there could be cases where group A and B will have different distributions but with the same means (such as two distributions, one with positive skewness and the other with a negative one, but shifted so to have the same means). In such cases, MW could have more than alpha level power in rejecting the Null hypothesis but attributing the interpretation of difference in means to such a result would be incorrect.

In the presence of an outlier, the t-test is not robust. For example, for two independent samples when the data distributions are asymmetric (that is, the distributions are skewed) or the distributions have large tails, then the Wilcoxon rank-sum test (also known as the Mann–Whitney U test) can have three to four times higher power than the t-test.[25][27][28] The nonparametric counterpart to the paired samples t-test is the Wilcoxon signed-rank test for paired samples. For a discussion on choosing between the t-test and nonparametric alternatives, see Lumley, et al. (2002).[17]

One-way analysis of variance (ANOVA) generalizes the two-sample t-test when the data belong to more than two groups.

When both paired observations and independent observations are present in the two sample design, assuming data are missing completely at random (MCAR), the paired observations or independent observations may be discarded in order to proceed with the standard tests above. Alternatively making use of all of the available data, assuming normality and MCAR, the generalized partially overlapping samples t-test could be used.[29]

A generalization of Student's t statistic, called Hotelling's t-squared statistic, allows for the testing of hypotheses on multiple (often correlated) measures within the same sample. For instance, a researcher might submit a number of subjects to a personality test consisting of multiple personality scales (e.g. the Minnesota Multiphasic Personality Inventory). Because measures of this type are usually positively correlated, it is not advisable to conduct separate univariate t-tests to test hypotheses, as these would neglect the covariance among measures and inflate the chance of falsely rejecting at least one hypothesis (Type I error). In this case a single multivariate test is preferable for hypothesis testing. Fisher's Method for combining multiple tests with alpha reduced for positive correlation among tests is one. Another is Hotelling's T² statistic follows a T² distribution. However, in practice the distribution is rarely used, since tabulated values for T² are hard to find. Usually, T² is converted instead to an F statistic.

For a one-sample multivariate test, the hypothesis is that the mean vector (μ) is equal to a given vector (μ₀). The test statistic is Hotelling's t²:

${\displaystyle t^{2}=n({\bar {\mathbf {x} }}-{{\boldsymbol {\mu }}_{0}})'{\mathbf {S} }^{-1}({\bar {\mathbf {x} }}-{{\boldsymbol {\mu }}_{0}})}$

where n is the sample size, x is the vector of column means and S is an m × m sample covariance matrix.

For a two-sample multivariate test, the hypothesis is that the mean vectors (μ₁, μ₂) of two samples are equal. The test statistic is Hotelling's two-sample t²:

${\displaystyle t^{2}={\frac {n_{1}n_{2}}{n_{1}+n_{2}}}\left({\bar {\mathbf {x} }}_{1}-{\bar {\mathbf {x} }}_{2}\right)'{\mathbf {S} _{\text{pooled}}}^{-1}\left({\bar {\mathbf {x} }}_{1}-{\bar {\mathbf {x} }}_{2}\right).}$

The two-sample t-test is a special case of simple linear regression as illustrated by the following example.

A clinical trial examines 6 patients given drug or placebo. 3 patients get 0 units of drug (the placebo group). 3 patients get 1 unit of drug (the active treatment group). At the end of treatment, the researchers measure the change from baseline in the number of words that each patient can recall in a memory test.

Data and code are given for the analysis using the R programming language with the t.test and lmfunctions for the t-test and linear regression. Here are the (fictitious) data generated in R.

> word.recall.data=data.frame(drug.dose=c(0,0,0,1,1,1), word.recall=c(1,2,3,5,6,7))

Perform the t-test. Notice that the assumption of equal variance, var.equal=T, is required to make the analysis exactly equivalent to simple linear regression.

> with(word.recall.data, t.test(word.recall~drug.dose, var.equal=T))

Running the R code gives the following results.

• The mean word.recall in the 0 drug.dose group is 2.
• The mean word.recall in the 1 drug.dose group is 6.
• The difference between treatment groups in the mean word.recall is 6 – 2 = 4.
• The difference in word.recall between drug doses is significant (p=0.00805).

Perform a linear regression of the same data. Calculations may be performed using the R function lm() for a linear model.

> word.recall.data.lm =  lm(word.recall~drug.dose, data=word.recall.data)
> summary(word.recall.data.lm)

The linear regression provides a table of coefficients and p-values.

The table of coefficients gives the following results.

• The estimate value of 2 for the intercept is the mean value of the word recall when the drug dose is 0.
• The estimate value of 4 for the drug dose indicates that for a 1-unit change in drug dose (from 0 to 1) there is a 4-unit change in mean word recall (from 2 to 6). This is the slope of the line joining the two group means.
• The p-value that the slope of 4 is different from 0 is p = 0.00805.

The coefficients for the linear regression specify the slope and intercept of the line that joins the two group means, as illustrated in the graph. The intercept is 2 and the slope is 4.

Compare the result from the linear regression to the result from the t-test.

• From the t-test, the difference between the group means is 6-2=4.
• From the regression, the slope is also 4 indicating that a 1-unit change in drug dose (from 0 to 1) gives a 4-unit change in mean word recall (from 2 to 6).
• The t-test p-value for the difference in means, and the regression p-value for the slope, are both 0.00805. The methods give identical results.

This example shows that, for the special case of a simple linear regression where there is a single x-variable that has values 0 and 1, the t-test gives the same results as the linear regression. The relationship can also be shown algebraically.

Recognizing this relationship between the t-test and linear regression facilitates the use of multiple linear regression and multi-way analysis of variance. These alternatives to t-tests allow for the inclusion of additional explanatory variables that are associated with the response. Including such additional explanatory variables using regression or anova reduces the otherwise unexplained variance, and commonly yields greater power to detect differences than do two-sample t-tests.