Akaike information criterion

As an example of a hypothesis test, consider the t-test to compare the means of two normally-distributed populations. The input to the t-test comprises a random sample from each of the two populations.

To formulate the test as a comparison of models, we construct two different models. The first model models the two populations as having potentially different means and standard deviations. The likelihood function for the first model is thus the product of the likelihoods for two distinct normal distributions; so it has four parameters: μ₁, σ₁, μ₂, σ₂. To be explicit, the likelihood function is as follows (denoting the sample sizes by n₁ and n₂).

${\displaystyle {\mathcal {L}}(\mu _{1},\sigma _{1},\mu _{2},\sigma _{2})\,=\,}$

${\displaystyle \;\;\;\;\;\;\;\;\prod _{i=1}^{n_{1}}{\frac {1}{{\sqrt {2\pi }}\sigma _{1}}}\exp \left(-{\frac {(x_{i}-\mu _{1})^{2}}{2\sigma _{1}^{2}}}\right)\;\,{\boldsymbol {\cdot }}\,\prod _{i=n_{1}+1}^{n_{1}+n_{2}}{\frac {1}{{\sqrt {2\pi }}\sigma _{2}}}\exp \left(-{\frac {(x_{i}-\mu _{2})^{2}}{2\sigma _{2}^{2}}}\right)}$

The second model models the two populations as having the same means but potentially different standard deviations. The likelihood function for the second model thus sets μ₁ = μ₂ in the above equation; so it has three parameters.

We then maximize the likelihood functions for the two models (in practice, we maximize the log-likelihood functions); after that, it is easy to calculate the AIC values of the models. We next calculate the relative likelihood. For instance, if the second model was only 0.01 times as likely as the first model, then we would omit the second model from further consideration: so we would conclude that the two populations have different means.

The t-test assumes that the two populations have identical standard deviations; the test tends to be unreliable if the assumption is false and the sizes of the two samples are very different (Welch's t-test would be better). Comparing the means of the populations via AIC, as in the example above, has an advantage by not making such assumptions.

For another example of a hypothesis test, suppose that we have two populations, and each member of each population is in one of two categories—category #1 or category #2. Each population is binomially distributed. We want to know whether the distributions of the two populations are the same. We are given a random sample from each of the two populations.

Let m be the size of the sample from the first population. Let m₁ be the number of observations (in the sample) in category #1; so the number of observations in category #2 is m − m₁. Similarly, let n be the size of the sample from the second population. Let n₁ be the number of observations (in the sample) in category #1.

Let p be the probability that a randomly-chosen member of the first population is in category #1. Hence, the probability that a randomly-chosen member of the first population is in category #2 is 1 − p. Note that the distribution of the first population has one parameter. Let q be the probability that a randomly-chosen member of the second population is in category #1. Note that the distribution of the second population also has one parameter.

To compare the distributions of the two populations, we construct two different models. The first model models the two populations as having potentially different distributions. The likelihood function for the first model is thus the product of the likelihoods for two distinct binomial distributions; so it has two parameters: p, q. To be explicit, the likelihood function is as follows.

${\displaystyle {\mathcal {L}}(p,q)\,=\,{\frac {m!}{m_{1}!(m-m_{1})!}}p^{m_{1}}(1-p)^{m-m_{1}}\;\,{\boldsymbol {\cdot }}\;\;{\frac {n!}{n_{1}!(n-n_{1})!}}q^{n_{1}}(1-q)^{n-n_{1}}}$

The second model models the two populations as having the same distribution. The likelihood function for the second model thus sets p = q in the above equation; so the second model has one parameter.

We then maximize the likelihood functions for the two models (in practice, we maximize the log-likelihood functions); after that, it is easy to calculate the AIC values of the models. We next calculate the relative likelihood. For instance, if the second model was only 0.01 times as likely as the first model, then we would omit the second model from further consideration: so we would conclude that the two populations have different distributions.

A statistical model must account for random errors. A straight line model might be formally described as y_i = b₀ + b₁x_i + ε_i. Here, the ε_i are the residuals from the straight line fit. If the ε_i are assumed to be i.i.d. Gaussian (with zero mean), then the model has three parameters: b₀, b₁, and the variance of the Gaussian distributions. Thus, when calculating the AIC value of this model, we should use k=3. More generally, for any least squares model with i.i.d. Gaussian residuals, the variance of the residuals' distributions should be counted as one of the parameters.[27]

As another example, consider a first-order autoregressive model, defined by x_i = c + φx_i−1 + ε_i, with the ε_i being i.i.d. Gaussian (with zero mean). For this model, there are three parameters: c, φ, and the variance of the ε_i. More generally, a pth-order autoregressive model has p + 2 parameters. (If, however, c is not estimated from the data, but instead given in advance, then there are only p + 1 parameters.)

The AIC values of the candidate models must all be computed with the same data set. Sometimes, though, we might want to compare a model of the response variable, y, with a model of the logarithm of the response variable, log(y). More generally, we might want to compare a model of the data with a model of transformed data. Following is an illustration of how to deal with data transforms (adapted from Burnham & Anderson (2002, §2.11.3): "Investigators should be sure that all hypotheses are modeled using the same response variable").

Suppose that we want to compare two models: one with a normal distribution of y and one with a normal distribution of log(y). We should not directly compare the AIC values of the two models. Instead, we should transform the normal cumulative distribution function to first take the logarithm of y. To do that, we need to perform the relevant integration by substitution: thus, we need to multiply by the derivative of the (natural) logarithm function, which is 1/y. Hence, the transformed distribution has the following probability density function:

${\displaystyle y\mapsto \,{\frac {1}{y}}{\frac {1}{\sqrt {2\pi \sigma ^{2}}}}\,\exp \left(-{\frac {\left(\ln y-\mu \right)^{2}}{2\sigma ^{2}}}\right)}$

—which is the probability density function for the log-normal distribution. We then compare the AIC value of the normal model against the AIC value of the log-normal model.

The formula for the Bayesian information criterion (BIC) is similar to the formula for AIC, but with a different penalty for the number of parameters. With AIC the penalty is 2k, whereas with BIC the penalty is ln(n)k.

A comparison of AIC/AICc and BIC is given by Burnham & Anderson (2002, §6.3-6.4), with follow-up remarks by Burnham & Anderson (2004). The authors show that AIC/AICc can be derived in the same Bayesian framework as BIC, just by using different prior probabilities. In the Bayesian derivation of BIC, though, each candidate model has a prior probability of 1/R (where R is the number of candidate models). Additionally, the authors present a few simulation studies that suggest AICc tends to have practical/performance advantages over BIC.

A point made by several researchers is that AIC and BIC are appropriate for different tasks. In particular, BIC is argued to be appropriate for selecting the "true model" (i.e. the process that generated the data) from the set of candidate models, whereas AIC is not appropriate. To be specific, if the "true model" is in the set of candidates, then BIC will select the "true model" with probability 1, as n → ∞; in contrast, when selection is done via AIC, the probability can be less than 1.[31][32][33] Proponents of AIC argue that this issue is negligible, because the "true model" is virtually never in the candidate set. Indeed, it is a common aphorism in statistics that "all models are wrong"; hence the "true model" (i.e. reality) cannot be in the candidate set.

Another comparison of AIC and BIC is given by Vrieze (2012). Vrieze presents a simulation study—which allows the "true model" to be in the candidate set (unlike with virtually all real data). The simulation study demonstrates, in particular, that AIC sometimes selects a much better model than BIC even when the "true model" is in the candidate set. The reason is that, for finite n, BIC can have a substantial risk of selecting a very bad model from the candidate set. This reason can arise even when n is much larger than k². With AIC, the risk of selecting a very bad model is minimized.

If the "true model" is not in the candidate set, then the most that we can hope to do is select the model that best approximates the "true model". AIC is appropriate for finding the best approximating model, under certain assumptions.[31][32][33] (Those assumptions include, in particular, that the approximating is done with regard to information loss.)

Comparison of AIC and BIC in the context of regression is given by Yang (2005). In regression, AIC is asymptotically optimal for selecting the model with the least mean squared error, under the assumption that the "true model" is not in the candidate set. BIC is not asymptotically optimal under the assumption. Yang additionally shows that the rate at which AIC converges to the optimum is, in a certain sense, the best possible.

Sometimes, each candidate model assumes that the residuals are distributed according to independent identical normal distributions (with zero mean). That gives rise to least squares model fitting.

With least squares fitting, the maximum likelihood estimate for the variance of a model's residuals distributions is the reduced chi-squared statistic,

${\displaystyle {\hat {\sigma }}^{2}=\mathrm {RSS} /\nu }$,

where the residual sum of squares is

${\displaystyle \textstyle \mathrm {RSS} =\sum _{i=1}^{n}(y_{i}-f(x_{i};{\hat {\theta }}))^{2}}$

and ${\displaystyle \nu =n-m}$ is the degree of freedom (for n observations and m unknowns). Then, the maximum value of a model's log-likelihood function is (see Normal distribution#Log-likelihood):

${\displaystyle \ln({\hat {L}})=-{\frac {n}{2}}\ln(2\pi )-{\frac {n}{2}}\ln({\hat {\sigma }}^{2})-{\frac {1}{2{\hat {\sigma }}^{2}}}\mathrm {RSS} \,=\,-{\frac {n}{2}}\ln({\hat {\sigma }}^{2})+C}$

where C is a constant independent of the model, and dependent only on the particular data points, i.e. it does not change if the data does not change.

That gives:[34]

${\displaystyle \mathrm {AIC} =2k-2\ln({\hat {L}})=2k+n\ln({\hat {\sigma }}^{2})-2C}$

Because only differences in AIC are meaningful, the constant C can be ignored, which allows us to conveniently take the following for model comparisons:

${\displaystyle \Delta \mathrm {AIC} =2k+n\ln({\hat {\sigma }}^{2})}$

Note that if all the models have the same k, then selecting the model with minimum AIC is equivalent to selecting the model with minimum RSS—which is the usual objective of model selection based on least squares.

Leave-one-out cross-validation is asymptotically equivalent to AIC, for ordinary linear regression models.[35] Asymptotic equivalence to AIC also holds for mixed-effects models.[36]

Mallows's C_p is equivalent to AIC in the case of (Gaussian) linear regression.[37]