Chapman–Robbins bound

Let ${\displaystyle \Theta }$ be the set of parameters for a family of probability distributions ${\displaystyle \{\mu _{\theta }:\theta \in \Theta \}}$ on ${\displaystyle \Omega }$.

For any two ${\displaystyle \theta ,\theta '\in \Theta }$, let ${\displaystyle \chi ^{2}(\mu _{\theta '};\mu _{\theta })}$ be the ${\displaystyle \chi ^{2}}$-divergence from ${\displaystyle \mu _{\theta }}$ to ${\displaystyle \mu _{\theta '}}$. Then:

A generalization to the multivariable case is:[3]

By the variational representation of chi-squared divergence:[3]

$${\displaystyle \chi ^{2}(P;Q)=\sup _{g}{\frac {(E_{P}[g]-E_{Q}[g])^{2}}{\operatorname {Var} _{Q}[g]}}}$$

Plug in ${\displaystyle g={\hat {g}},P=\mu _{\theta '},Q=\mu _{\theta }}$, to obtain:

$${\displaystyle \chi ^{2}(\mu _{\theta '};\mu _{\theta })\geq {\frac {(E_{\theta '}[{\hat {g}}]-E_{\theta }[{\hat {g}}])^{2}}{\operatorname {Var} _{\theta }[{\hat {g}}]}}}$$

Switch the denominator and the left side and take supremum over ${\displaystyle \theta '}$ to obtain the single-variate case. For the multivariate case, we define ${\textstyle h=\sum _{i=1}^{m}v_{i}{\hat {g}}_{i}}$ for any ${\displaystyle v\neq 0\in \mathbb {R} ^{m}}$. Then plug in ${\displaystyle g=h}$ in the variational representation to obtain:

$${\displaystyle \chi ^{2}(\mu _{\theta '};\mu _{\theta })\geq {\frac {(E_{\theta '}[h]-E_{\theta }[h])^{2}}{\operatorname {Var} _{\theta }[h]}}={\frac {\langle v,E_{\theta '}[{\hat {g}}]-E_{\theta }[{\hat {g}}]\rangle ^{2}}{v^{T}\operatorname {Cov} _{\theta }[{\hat {g}}]v}}}$$

Take supremum over ${\displaystyle v\neq 0\in \mathbb {R} ^{m}}$, using the linear algebra fact that ${\displaystyle \sup _{v\neq 0}{\frac {v^{T}ww^{T}v}{v^{T}Mv}}=w^{T}M^{-1}w}$, we obtain the multivariate case.

Usually, ${\displaystyle \Omega ={\mathcal {X}}^{n}}$ is the sample space of ${\displaystyle n}$ independent draws of a ${\displaystyle {\mathcal {X}}}$-valued random variable ${\displaystyle X}$ with distribution ${\displaystyle \lambda _{\theta }}$ from a by ${\displaystyle \theta \in \Theta \subseteq \mathbb {R} ^{m}}$ parameterized family of probability distributions, ${\displaystyle \mu _{\theta }=\lambda _{\theta }^{\otimes n}}$ is its ${\displaystyle n}$-fold product measure, and ${\displaystyle {\hat {g}}:{\mathcal {X}}^{n}\to \Theta }$ is an estimator of ${\displaystyle \theta }$. Then, for ${\displaystyle m=1}$, the expression inside the supremum in the Chapman–Robbins bound converges to the Cramér–Rao bound of ${\displaystyle {\hat {g}}}$ when ${\displaystyle \theta '\to \theta }$, assuming the regularity conditions of the Cramér–Rao bound hold. This implies that, when both bounds exist, the Chapman–Robbins version is always at least as tight as the Cramér–Rao bound; in many cases, it is substantially tighter.

The Chapman–Robbins bound also holds under much weaker regularity conditions. For example, no assumption is made regarding differentiability of the probability density function p(x; θ) of ${\displaystyle \lambda _{\theta }}$. When p(x; θ) is non-differentiable, the Fisher information is not defined, and hence the Cramér–Rao bound does not exist.

• Cramér–Rao bound
• Estimation theory

• Lehmann, E. L.; Casella, G. (1998), Theory of Point Estimation (2nd ed.), Springer, pp. 113–114, ISBN 0-387-98502-6