Fisher information

The regularity conditions are as follows:[9]

1. The partial derivative of f(X; θ) with respect to θ exists almost everywhere. (It can fail to exist on a null set, as long as this set does not depend on θ.)
2. The integral of f(X; θ) can be differentiated under the integral sign with respect to θ.
3. The support of f(X; θ) does not depend on θ.

If θ is a vector then the regularity conditions must hold for every component of θ. It is easy to find an example of a density that does not satisfy the regularity conditions: The density of a Uniform(0, θ) variable fails to satisfy conditions 1 and 3. In this case, even though the Fisher information can be computed from the definition, it will not have the properties it is typically assumed to have.

Because the likelihood of θ given X is always proportional to the probability f(X; θ), their logarithms necessarily differ by a constant that is independent of θ, and the derivatives of these logarithms with respect to θ are necessarily equal. Thus one can substitute in a log-likelihood l(θ; X) instead of log f(X; θ) in the definitions of Fisher Information.

The value X can represent a single sample drawn from a single distribution or can represent a collection of samples drawn from a collection of distributions. If there are n samples and the corresponding n distributions are statistically independent then the Fisher information will necessarily be the sum of the single-sample Fisher information values, one for each single sample from its distribution. In particular, if the n distributions are independent and identically distributed then the Fisher information will necessarily be n times the Fisher information of a single sample from the common distribution.

The Cramér–Rao bound[10][11] states that the inverse of the Fisher information is a lower bound on the variance of any unbiased estimator of θ. H.L. Van Trees (1968) and B. Roy Frieden (2004) provide the following method of deriving the Cramér–Rao bound, a result which describes use of the Fisher information.

Informally, we begin by considering an unbiased estimator ${\displaystyle {\hat {\theta }}(X)}$. Mathematically, "unbiased" means that

${\displaystyle \operatorname {E} \left[\left.{\hat {\theta }}(X)-\theta \,\,\right|\,\,\theta \right]=\int \left({\hat {\theta }}(x)-\theta \right)\,f(x;\theta )\,dx=0{\text{ regardless of the value of }}\theta .}$

This expression is zero independent of θ, so its partial derivative with respect to θ must also be zero. By the product rule, this partial derivative is also equal to

${\displaystyle 0={\frac {\partial }{\partial \theta }}\int \left({\hat {\theta }}(x)-\theta \right)\,f(x;\theta )\,dx=\int \left({\hat {\theta }}(x)-\theta \right){\frac {\partial f}{\partial \theta }}\,dx-\int f\,dx.}$

For each θ, the likelihood function is a probability density function, and therefore ${\displaystyle \int f\,dx=1}$. By using the chain rule on the partial derivative of ${\displaystyle \log f}$ and then dividing and multiplying by ${\displaystyle f(x;\theta )}$, one can verify that

${\displaystyle {\frac {\partial f}{\partial \theta }}=f\,{\frac {\partial \log f}{\partial \theta }}.}$

Using these two facts in the above, we get

${\displaystyle \int \left({\hat {\theta }}-\theta \right)f\,{\frac {\partial \log f}{\partial \theta }}\,dx=1.}$

Factoring the integrand gives

${\displaystyle \int \left(\left({\hat {\theta }}-\theta \right){\sqrt {f}}\right)\left({\sqrt {f}}\,{\frac {\partial \log f}{\partial \theta }}\right)\,dx=1.}$

Squaring the expression in the integral, the Cauchy–Schwarz inequality yields

${\displaystyle 1={\biggl (}\int \left[\left({\hat {\theta }}-\theta \right){\sqrt {f}}\right]\cdot \left[{\sqrt {f}}\,{\frac {\partial \log f}{\partial \theta }}\right]\,dx{\biggr )}^{2}\leq \left[\int \left({\hat {\theta }}-\theta \right)^{2}f\,dx\right]\cdot \left[\int \left({\frac {\partial \log f}{\partial \theta }}\right)^{2}f\,dx\right].}$

The second bracketed factor is defined to be the Fisher Information, while the first bracketed factor is the expected mean-squared error of the estimator ${\displaystyle {\hat {\theta }}}$. By rearranging, the inequality tells us that

${\displaystyle \operatorname {Var} \left({\hat {\theta }}\right)\geq {\frac {1}{{\mathcal {I}}\left(\theta \right)}}.}$

In other words, the precision to which we can estimate θ is fundamentally limited by the Fisher information of the likelihood function.

Alternatively, the same conclusion can be obtained directly from the Cauchy–Schwarz inequality for random variables, ${\displaystyle |\operatorname {Cov} (A,B)|^{2}\leq \operatorname {Var} (A)\operatorname {Var} (B)}$, applied to the random variables ${\displaystyle {\hat {\theta }}(X)}$ and ${\displaystyle \partial _{\theta }\log f(X;\theta )}$, and observing that for unbiased estimators we have

$${\displaystyle \operatorname {Cov} [{\hat {\theta }}(X),\partial _{\theta }\log f(X;\theta )]=\int {\hat {\theta }}(x)\,\partial _{\theta }f(x;\theta )\,dx=\partial _{\theta }\operatorname {E} [{\hat {\theta }}]=1.}$$

A Bernoulli trial is a random variable with two possible outcomes, "success" and "failure", with success having a probability of θ. The outcome can be thought of as determined by a coin toss, with the probability of heads being θ and the probability of tails being 1 − θ.

Let X be a Bernoulli trial. The Fisher information contained in X may be calculated to be

${\displaystyle {\begin{aligned}{\mathcal {I}}(\theta )&=-\operatorname {E} \left[\left.{\frac {\partial ^{2}}{\partial \theta ^{2}}}\log \left(\theta ^{X}(1-\theta )^{1-X}\right)\right|\theta \right]\\[5pt]&=-\operatorname {E} \left[\left.{\frac {\partial ^{2}}{\partial \theta ^{2}}}\left(X\log \theta +(1-X)\log(1-\theta )\right)\,\,\right|\,\,\theta \right]\\[5pt]&=\operatorname {E} \left[\left.{\frac {X}{\theta ^{2}}}+{\frac {1-X}{(1-\theta )^{2}}}\,\,\right|\,\,\theta \right]\\[5pt]&={\frac {\theta }{\theta ^{2}}}+{\frac {1-\theta }{(1-\theta )^{2}}}\\[5pt]&={\frac {1}{\theta (1-\theta )}}.\end{aligned}}}$

Because Fisher information is additive, the Fisher information contained in n independent Bernoulli trials is therefore

${\displaystyle {\mathcal {I}}(\theta )={\frac {n}{\theta (1-\theta )}}.}$

This is the reciprocal of the variance of the mean number of successes in n Bernoulli trials, so in this case, the Cramér–Rao bound is an equality.

We say that two parameter component vectors θ₁ and θ₂ are information orthogonal if the Fisher information matrix is block diagonal, with these components in separate blocks.[16] Orthogonal parameters are easy to deal with in the sense that their maximum likelihood estimates are asymptotically uncorrelated. When considering how to analyse a statistical model, the modeller is advised to invest some time searching for an orthogonal parametrization of the model, in particular when the parameter of interest is one-dimensional, but the nuisance parameter can have any dimension.[17]

If the Fisher information matrix is positive definite for all θ, then the corresponding statistical model is said to be regular; otherwise, the statistical model is said to be singular.[18] Examples of singular statistical models include the following: normal mixtures, binomial mixtures, multinomial mixtures, Bayesian networks, neural networks, radial basis functions, hidden Markov models, stochastic context-free grammars, reduced rank regressions, Boltzmann machines.

In machine learning, if a statistical model is devised so that it extracts hidden structure from a random phenomenon, then it naturally becomes singular.[19]

The FIM for a N-variate multivariate normal distribution, ${\displaystyle \,X\sim N\left(\mu (\theta ),\,\Sigma (\theta )\right)}$ has a special form. Let the K-dimensional vector of parameters be ${\displaystyle \theta ={\begin{bmatrix}\theta _{1}&\dots &\theta _{K}\end{bmatrix}}^{\textsf {T}}}$ and the vector of random normal variables be ${\displaystyle X={\begin{bmatrix}X_{1}&\dots &X_{N}\end{bmatrix}}^{\textsf {T}}}$. Assume that the mean values of these random variables are ${\displaystyle \,\mu (\theta )={\begin{bmatrix}\mu _{1}(\theta )&\dots &\mu _{N}(\theta )\end{bmatrix}}^{\textsf {T}}}$, and let ${\displaystyle \,\Sigma (\theta )}$ be the covariance matrix. Then, for ${\displaystyle 1\leq m,\,n\leq K}$, the (m, n) entry of the FIM is:[20]

${\displaystyle {\mathcal {I}}_{m,n}={\frac {\partial \mu ^{\textsf {T}}}{\partial \theta _{m}}}\Sigma ^{-1}{\frac {\partial \mu }{\partial \theta _{n}}}+{\frac {1}{2}}\operatorname {tr} \left(\Sigma ^{-1}{\frac {\partial \Sigma }{\partial \theta _{m}}}\Sigma ^{-1}{\frac {\partial \Sigma }{\partial \theta _{n}}}\right),}$

where ${\displaystyle (\cdot )^{\textsf {T}}}$ denotes the transpose of a vector, ${\displaystyle \operatorname {tr} (\cdot )}$ denotes the trace of a square matrix, and:

${\displaystyle {\begin{aligned}{\frac {\partial \mu }{\partial \theta _{m}}}&={\begin{bmatrix}{\dfrac {\partial \mu _{1}}{\partial \theta _{m}}}&{\dfrac {\partial \mu _{2}}{\partial \theta _{m}}}&\cdots &{\dfrac {\partial \mu _{N}}{\partial \theta _{m}}}\end{bmatrix}}^{\textsf {T}};\\[8pt]{\dfrac {\partial \Sigma }{\partial \theta _{m}}}&={\begin{bmatrix}{\dfrac {\partial \Sigma _{1,1}}{\partial \theta _{m}}}&{\dfrac {\partial \Sigma _{1,2}}{\partial \theta _{m}}}&\cdots &{\dfrac {\partial \Sigma _{1,N}}{\partial \theta _{m}}}\\[5pt]{\dfrac {\partial \Sigma _{2,1}}{\partial \theta _{m}}}&{\dfrac {\partial \Sigma _{2,2}}{\partial \theta _{m}}}&\cdots &{\dfrac {\partial \Sigma _{2,N}}{\partial \theta _{m}}}\\\vdots &\vdots &\ddots &\vdots \\{\dfrac {\partial \Sigma _{N,1}}{\partial \theta _{m}}}&{\dfrac {\partial \Sigma _{N,2}}{\partial \theta _{m}}}&\cdots &{\dfrac {\partial \Sigma _{N,N}}{\partial \theta _{m}}}\end{bmatrix}}.\end{aligned}}}$

Note that a special, but very common, case is the one where ${\displaystyle \Sigma (\theta )=\Sigma }$, a constant. Then

${\displaystyle {\mathcal {I}}_{m,n}={\frac {\partial \mu ^{\textsf {T}}}{\partial \theta _{m}}}\Sigma ^{-1}{\frac {\partial \mu }{\partial \theta _{n}}}.\ }$

In this case the Fisher information matrix may be identified with the coefficient matrix of the normal equations of least squares estimation theory.

Another special case occurs when the mean and covariance depend on two different vector parameters, say, β and θ. This is especially popular in the analysis of spatial data, which often uses a linear model with correlated residuals. In this case,[21]

${\displaystyle {\mathcal {I}}(\beta ,\theta )=\operatorname {diag} \left({\mathcal {I}}(\beta ),{\mathcal {I}}(\theta )\right)}$

where

${\displaystyle {\begin{aligned}{\mathcal {I}}{(\beta )_{m,n}}&={\frac {\partial \mu ^{\textsf {T}}}{\partial \beta _{m}}}\Sigma ^{-1}{\frac {\partial \mu }{\partial \beta _{n}}},\\[5pt]{\mathcal {I}}{(\theta )_{m,n}}&={\frac {1}{2}}\operatorname {tr} \left(\Sigma ^{-1}{\frac {\partial \Sigma }{\partial \theta _{m}}}{\Sigma ^{-1}}{\frac {\partial \Sigma }{\partial \theta _{n}}}\right)\end{aligned}}}$

Similar to the entropy or mutual information, the Fisher information also possesses a chain rule decomposition. In particular, if X and Y are jointly distributed random variables, it follows that:[22]

${\displaystyle {\mathcal {I}}_{X,Y}(\theta )={\mathcal {I}}_{X}(\theta )+{\mathcal {I}}_{Y\mid X}(\theta ),}$

where ${\displaystyle {\mathcal {I}}_{Y\mid X}(\theta )=\operatorname {E} _{X}\left[{\mathcal {I}}_{Y\mid X=x}(\theta )\right]}$ and ${\displaystyle {\mathcal {I}}_{Y\mid X=x}(\theta )}$ is the Fisher information of Y relative to ${\displaystyle \theta }$ calculated with respect to the conditional density of Y given a specific value X = x.

As a special case, if the two random variables are independent, the information yielded by the two random variables is the sum of the information from each random variable separately:

${\displaystyle {\mathcal {I}}_{X,Y}(\theta )={\mathcal {I}}_{X}(\theta )+{\mathcal {I}}_{Y}(\theta ).}$

Consequently, the information in a random sample of n independent and identically distributed observations is n times the information in a sample of size 1.

Given a convex function ${\displaystyle f:[0,\infty )\to (-\infty ,\infty ]}$ that ${\displaystyle f(x)}$ is finite for all ${\displaystyle x>0}$, ${\displaystyle f(1)=0}$, and ${\displaystyle f(0)=\lim _{t\to 0^{+}}f(t)}$, (which could be infinite), it defines an f-divergence ${\displaystyle D_{f}}$. Then if ${\displaystyle f}$ is strictly convex at ${\displaystyle 1}$, then locally at ${\displaystyle \theta \in \Theta }$, the Fisher information matrix is a metric, in the sense that[23]

$${\displaystyle (\delta \theta )^{T}I(\theta )(\delta \theta )={\frac {1}{f''(1)}}D_{f}(P_{\theta +\delta \theta }\parallel P_{\theta })}$$

where ${\displaystyle P_{\theta }}$ is the distribution parametrized by ${\displaystyle \theta }$. That is, it's the distribution with pdf ${\displaystyle f(x;\theta )}$.

In this form, it is clear that the Fisher information matrix is a Riemannian metric, and varies correctly under a change of variables. (see section on Reparametrization.)

The information provided by a sufficient statistic is the same as that of the sample X. This may be seen by using Neyman's factorization criterion for a sufficient statistic. If T(X) is sufficient for θ, then

${\displaystyle f(X;\theta )=g(T(X),\theta )h(X)}$

for some functions g and h. The independence of h(X) from θ implies

${\displaystyle {\frac {\partial }{\partial \theta }}\log \left[f(X;\theta )\right]={\frac {\partial }{\partial \theta }}\log \left[g(T(X);\theta )\right],}$

and the equality of information then follows from the definition of Fisher information. More generally, if T = t(X) is a statistic, then

${\displaystyle {\mathcal {I}}_{T}(\theta )\leq {\mathcal {I}}_{X}(\theta )}$

with equality if and only if T is a sufficient statistic.[24]

The Fisher information depends on the parametrization of the problem. If θ and η are two scalar parametrizations of an estimation problem, and θ is a continuously differentiable function of η, then

${\displaystyle {\mathcal {I}}_{\eta }(\eta )={\mathcal {I}}_{\theta }(\theta (\eta ))\left({\frac {d\theta }{d\eta }}\right)^{2}}$

where ${\displaystyle {\mathcal {I}}_{\eta }}$ and ${\displaystyle {\mathcal {I}}_{\theta }}$ are the Fisher information measures of η and θ, respectively.[25]

In the vector case, suppose ${\displaystyle {\boldsymbol {\theta }}}$ and ${\displaystyle {\boldsymbol {\eta }}}$ are k-vectors which parametrize an estimation problem, and suppose that ${\displaystyle {\boldsymbol {\theta }}}$ is a continuously differentiable function of ${\displaystyle {\boldsymbol {\eta }}}$, then,[26]

${\displaystyle {\mathcal {I}}_{\boldsymbol {\eta }}({\boldsymbol {\eta }})={\boldsymbol {J}}^{\textsf {T}}{\mathcal {I}}_{\boldsymbol {\theta }}({\boldsymbol {\theta }}({\boldsymbol {\eta }})){\boldsymbol {J}}}$

where the (i, j)th element of the k × k Jacobian matrix ${\displaystyle {\boldsymbol {J}}}$ is defined by

${\displaystyle J_{ij}={\frac {\partial \theta _{i}}{\partial \eta _{j}}},}$

and where ${\displaystyle {\boldsymbol {J}}^{\textsf {T}}}$ is the matrix transpose of ${\displaystyle {\boldsymbol {J}}.}$

In information geometry, this is seen as a change of coordinates on a Riemannian manifold, and the intrinsic properties of curvature are unchanged under different parametrizations. In general, the Fisher information matrix provides a Riemannian metric (more precisely, the Fisher–Rao metric) for the manifold of thermodynamic states, and can be used as an information-geometric complexity measure for a classification of phase transitions, e.g., the scalar curvature of the thermodynamic metric tensor diverges at (and only at) a phase transition point.[27]

In the thermodynamic context, the Fisher information matrix is directly related to the rate of change in the corresponding order parameters.[28] In particular, such relations identify second-order phase transitions via divergences of individual elements of the Fisher information matrix.

The Fisher information matrix plays a role in an inequality like the isoperimetric inequality.[29] Of all probability distributions with a given entropy, the one whose Fisher information matrix has the smallest trace is the Gaussian distribution. This is like how, of all bounded sets with a given volume, the sphere has the smallest surface area.

The proof involves taking a multivariate random variable ${\displaystyle X}$ with density function ${\displaystyle f}$ and adding a location parameter to form a family of densities ${\displaystyle \{f(x-\theta )\mid \theta \in \mathbb {R} ^{n}\}}$. Then, by analogy with the Minkowski–Steiner formula, the "surface area" of ${\displaystyle X}$ is defined to be

${\displaystyle S(X)=\lim _{\varepsilon \to 0}{\frac {e^{H(X+Z_{\varepsilon })}-e^{H(X)}}{\varepsilon }}}$

where ${\displaystyle Z_{\varepsilon }}$ is a Gaussian variable with covariance matrix ${\displaystyle \varepsilon I}$. The name "surface area" is apt because the entropy power ${\displaystyle e^{H(X)}}$ is the volume of the "effective support set,"[30] so ${\displaystyle S(X)}$ is the "derivative" of the volume of the effective support set, much like the Minkowski-Steiner formula. The remainder of the proof uses the entropy power inequality, which is like the Brunn–Minkowski inequality. The trace of the Fisher information matrix is found to be a factor of ${\displaystyle S(X)}$.

Fisher information is widely used in optimal experimental design. Because of the reciprocity of estimator-variance and Fisher information, minimizing the variance corresponds to maximizing the information.

When the linear (or linearized) statistical model has several parameters, the mean of the parameter estimator is a vector and its variance is a matrix. The inverse of the variance matrix is called the "information matrix". Because the variance of the estimator of a parameter vector is a matrix, the problem of "minimizing the variance" is complicated. Using statistical theory, statisticians compress the information-matrix using real-valued summary statistics; being real-valued functions, these "information criteria" can be maximized.

Traditionally, statisticians have evaluated estimators and designs by considering some summary statistic of the covariance matrix (of an unbiased estimator), usually with positive real values (like the determinant or matrix trace). Working with positive real numbers brings several advantages: If the estimator of a single parameter has a positive variance, then the variance and the Fisher information are both positive real numbers; hence they are members of the convex cone of nonnegative real numbers (whose nonzero members have reciprocals in this same cone).

For several parameters, the covariance matrices and information matrices are elements of the convex cone of nonnegative-definite symmetric matrices in a partially ordered vector space, under the Loewner (Löwner) order. This cone is closed under matrix addition and inversion, as well as under the multiplication of positive real numbers and matrices. An exposition of matrix theory and Loewner order appears in Pukelsheim.[31]

The traditional optimality criteria are the information matrix's invariants, in the sense of invariant theory; algebraically, the traditional optimality criteria are functionals of the eigenvalues of the (Fisher) information matrix (see optimal design).

In Bayesian statistics, the Fisher information is used to calculate the Jeffreys prior, which is a standard, non-informative prior for continuous distribution parameters.[32]

The Fisher information has been used to find bounds on the accuracy of neural codes. In that case, X is typically the joint responses of many neurons representing a low dimensional variable θ (such as a stimulus parameter). In particular the role of correlations in the noise of the neural responses has been studied.[33]

Fisher information plays a central role in a controversial principle put forward by Frieden as the basis of physical laws, a claim that has been disputed.[34]

The Fisher information is used in machine learning techniques such as elastic weight consolidation,[35] which reduces catastrophic forgetting in artificial neural networks.

Fisher information can be used as an alternative to the Hessian of the loss function in second-order gradient descent network training.[36]