p-value

Consider an observed test-statistic ${\displaystyle t}$ from unknown distribution ${\displaystyle T}$. Then the p-value ${\displaystyle p}$ is what the prior probability would be of observing a test-statistic value at least as "extreme" as ${\displaystyle t}$ if null hypothesis ${\displaystyle H_{0}}$ were true. That is:

• ${\displaystyle p=\Pr(T\geq t\mid H_{0})}$ for a one-sided right-tail test,
• ${\displaystyle p=\Pr(T\leq t\mid H_{0})}$ for a one-sided left-tail test,
• ${\displaystyle p=2\min\{\Pr(T\geq t\mid H_{0}),\Pr(T\leq t\mid H_{0})\}}$ for a two-sided test. If the distribution of ${\displaystyle T}$ is symmetric about zero, then ${\displaystyle p=\Pr(|T|\geq |t|\mid H_{0})}$

In a significance test, the null hypothesis ${\displaystyle H_{0}}$ is rejected if the p-value is less than or equal to a predefined threshold value ${\displaystyle \alpha }$, which is referred to as the alpha level or significance level. ${\displaystyle \alpha }$ is not derived from the data, but rather is set by the researcher before examining the data. ${\displaystyle \alpha }$ is commonly set to 0.05, though lower alpha levels are sometimes used.

The p-value is a function of the chosen test statistic ${\displaystyle T}$ and is therefore a random variable. If the null hypothesis fixes the probability distribution of ${\displaystyle T}$ precisely, and if that distribution is continuous, then when the null-hypothesis is true, the p-value is uniformly distributed between 0 and 1. Thus, the p-value is not fixed. If the same test is repeated independently with fresh data, one will typically obtain a different p-value in each iteration. If the null-hypothesis is composite, or the distribution of the statistic is discrete, the probability of obtaining a p-value less than or equal to any number between 0 and 1 is less than or equal to that number, if the null-hypothesis is true. It remains the case that very small values are relatively unlikely if the null-hypothesis is true, and that a significance test at level ${\displaystyle \alpha }$ is obtained by rejecting the null-hypothesis if the significance level is less than or equal to ${\displaystyle \alpha }$.

Different p-values based on independent sets of data can be combined, for instance using Fisher's combined probability test.

When the null hypothesis is true, if it takes the form ${\displaystyle H_{0}:\theta =\theta _{0}}$, and the underlying random variable is continuous, then the probability distribution of the p-value is uniform on the interval [0,1]. By contrast, if the alternative hypothesis is true, the distribution is dependent on sample size and the true value of the parameter being studied.[6][7]

The distribution of p-values for a group of studies is sometimes called a p-curve.[8] A p-curve can be used to assess the reliability of scientific literature, such as by detecting publication bias or p-hacking.[8][9]

In parametric hypothesis testing problems, a simple or point hypothesis refers to a hypothesis where the parameter's value is assumed to be a single number. In contrast, in a composite hypothesis the parameter's value is given by a set of numbers. For example, when testing the null hypothesis that a distribution is normal with a mean less than or equal to zero against the alternative that the mean is greater than zero (variance known), the null hypothesis does not specify the probability distribution of the appropriate test statistic. In the just mentioned example that would be the Z-statistic belonging to the one-sided one-sample Z-test. For each possible value of the theoretical mean, the Z-test statistic has a different probability distribution. In these circumstances (the case of a so-called composite null hypothesis) the p-value is defined by taking the least favourable null-hypothesis case, which is typically on the border between null and alternative.

This definition ensures the complementarity of p-values and alpha-levels. If we set the significance level alpha to 0.05, and only reject the null hypothesis if the p-value is less than or equal to 0.05, then our hypothesis test will indeed have significance level (maximal type 1 error rate) 0.05. As Neyman wrote: “The error that a practising statistician would consider the more important to avoid (which is a subjective judgment) is called the error of the first kind. The first demand of the mathematical theory is to deduce such test criteria as would ensure that the probability of committing an error of the first kind would equal (or approximately equal, or not exceed) a preassigned number α, such as α = 0.05 or 0.01, etc. This number is called the level of significance”; Neyman 1976, p. 161 in "The Emergence of Mathematical Statistics: A Historical Sketch with Particular Reference to the United States","On the History of Statistics and Probability", ed. D.B. Owen, New York: Marcel Dekker, pp. 149-193. See also "Confusion Over Measures of Evidence (p's) Versus Errors (a's) in Classical Statistical Testing", Raymond Hubbard and M. J. Bayarri, The American Statistician, August 2003, Vol. 57, No 3, 171--182 (with discussion). For a concise modern statement see Chapter 10 of "All of Statistics: A Concise Course in Statistical Inference", Springer; 1st Corrected ed. 20 edition (September 17, 2004). Larry Wasserman.

According to the ASA, there is widespread agreement that p-values are often misused and misinterpreted.[3] One practice that has been particularly criticized is accepting the alternative hypothesis for any p-value nominally less than .05 without other supporting evidence. Although p-values are helpful in assessing how incompatible the data are with a specified statistical model, contextual factors must also be considered, such as "the design of a study, the quality of the measurements, the external evidence for the phenomenon under study, and the validity of assumptions that underlie the data analysis".[3] Another concern is that the p-value is often misunderstood as being the probability that the null hypothesis is true.[3][11]

Some statisticians have proposed abandoning p-values and focusing more on other inferential statistics,[3] such as confidence intervals,[12][13] likelihood ratios,[14][15] or Bayes factors,[16][17][18] but there is heated debate on the feasibility of these alternatives.[19][20] Others have suggested to remove fixed significance thresholds and to interpret p-values as continuous indices of the strength of evidence against the null hypothesis.[21][22] Yet others suggested to report alongside p-values the prior probability of a real effect that would be required to obtain a false positive risk (i.e. the probability that there is no real effect) below a pre specified threshold (e.g. 5%).[23]

As an example of a statistical test, an experiment is performed to determine whether a coin flip is fair (equal chance of landing heads or tails) or unfairly biased (one outcome being more likely than the other).

Suppose that the experimental results show the coin turning up heads 14 times out of 20 total flips. The full data ${\displaystyle X}$ would be a sequence of twenty times the symbol "H" or "T". The statistic on which one might focus could be the total number ${\displaystyle T}$ of heads. The null hypothesis is that the coin is fair, and coin tosses are independent of one another. If a right-tailed test is considered, which would be the case if one is actually interested in the possibility that the coin is biased towards falling heads, then the p-value of this result is the chance of a fair coin landing on heads at least 14 times out of 20 flips. That probability can be computed from binomial coefficients as

${\displaystyle {\begin{aligned}&\Pr(14{\text{ heads}})+\Pr(15{\text{ heads}})+\cdots +\Pr(20{\text{ heads}})\\&={\frac {1}{2^{20}}}\left[{\binom {20}{14}}+{\binom {20}{15}}+\cdots +{\binom {20}{20}}\right]={\frac {60,\!460}{1,\!048,\!576}}\approx 0.058\end{aligned}}}$

This probability is the p-value, considering only extreme results that favor heads. This is called a one-tailed test. However, one might be interested in deviations in either direction, favoring either heads or tails. The two-tailed p-value, which considers deviations favoring either heads or tails, may instead be calculated. As the binomial distribution is symmetrical for a fair coin, the two-sided p-value is simply twice the above calculated single-sided p-value: the two-sided p-value is 0.115.

In the above example:

• Null hypothesis (H₀): The coin is fair, with Pr(heads) = 0.5
• Test statistic: Number of heads
• Alpha level (designated threshold of significance): 0.05
• Observation O: 14 heads out of 20 flips; and
• Two-tailed p-value of observation O given H₀ = 2 × min(Pr(no. of heads ≥ 14 heads), Pr(no. of heads ≤ 14 heads)) = 2 × min(0.058, 0.978) = 2*0.058 = 0.115.

Note that the Pr (no. of heads ≤ 14 heads) = 1 - Pr(no. of heads ≥ 14 heads) + Pr (no. of head = 14) = 1 - 0.058 + 0.036 = 0.978; however, the symmetry of this binomial distribution makes it an unnecessary computation to find the smaller of the two probabilities. Here, the calculated p-value exceeds .05, meaning that the data falls within the range of what would happen 95% of the time were the coin in fact fair. Hence, the null hypothesis is not rejected at the .05 level.

However, had one more head been obtained, the resulting p-value (two-tailed) would have been 0.0414 (4.14%), in which case the null hypothesis would be rejected at the .05 level.

The difference between the two meanings of "extreme" appear when we consider a multistage experiment for testing the fairness of the coin. Suppose we design the experiment as follows:

• Flip the coin twice. If both comes up heads or tails, end the experiment.
• Else, flip the coin 4 more times.

This experiment has 7 types of outcomes: 2 heads, 2 tails, 5 heads 1 tail..., 1 head 5 tails. We now calculate the p-value of the "3 heads 3 tails" outcome .

If we use the test statistic ${\displaystyle {\frac {heads}{tails}}}$, then under the null hypothesis is exactly 1 for two-sided p-value, and exactly ${\displaystyle {\frac {19}{32}}}$ for one-sided left-tail p-value, and same for one-sided right-tail p-value.

If we consider every outcome that has equal or lower probability than "3 heads 3 tails" as "at least as extreme", then the p-value is exactly ${\displaystyle {\frac {1}{2}}}$.

However, suppose we have planned to simply flip the coin 6 times no matter what happens, then the second definition of p-value would mean that the p-value of "3 heads 3 tails" is exactly 1.

Thus, the "at least as extreme" definition of p-value is deeply contextual, and depend on what the experimenter planned to do even in situations that did not occur.