Law of large numbers

Simulation illustrating the law of large numbers. Each frame, a coin that is red on one side and blue on the other is flipped, and a dot is added in the corresponding column. A pie chart shows the proportion of red and blue so far. Notice that while the proportion varies significantly at first, it approaches 50% as the number of trials increases.

The weak law of large numbers (also called Khinchin's law) states that the sample average converges in probability towards the expected value[17]

${\displaystyle {\begin{matrix}{}\\{\overline {X}}_{n}\ {\xrightarrow {P}}\ \mu \qquad {\textrm {when}}\ n\to \infty .\\{}\end{matrix}}}$ (2)

That is, for any positive number ε,

${\displaystyle \lim _{n\to \infty }\Pr \!\left(\,|{\overline {X}}_{n}-\mu |<\varepsilon \,\right)=1.}$

Interpreting this result, the weak law states that for any nonzero margin specified (ε), no matter how small, with a sufficiently large sample there will be a very high probability that the average of the observations will be close to the expected value; that is, within the margin.

As mentioned earlier, the weak law applies in the case of i.i.d. random variables, but it also applies in some other cases. For example, the variance may be different for each random variable in the series, keeping the expected value constant. If the variances are bounded, then the law applies, as shown by Chebyshev as early as 1867. (If the expected values change during the series, then we can simply apply the law to the average deviation from the respective expected values. The law then states that this converges in probability to zero.) In fact, Chebyshev's proof works so long as the variance of the average of the first n values goes to zero as n goes to infinity.[12] As an example, assume that each random variable in the series follows a Gaussian distribution with mean zero, but with variance equal to ${\displaystyle 2n/\log(n+1)}$, which is not bounded. At each stage, the average will be normally distributed (as the average of a set of normally distributed variables). The variance of the sum is equal to the sum of the variances, which is asymptotic to ${\displaystyle n^{2}/\log n}$. The variance of the average is therefore asymptotic to ${\displaystyle 1/\log n}$ and goes to zero.

There are also examples of the weak law applying even though the expected value does not exist.

The strong law of large numbers (also called Kolmogorov's law) states that the sample average converges almost surely to the expected value[18]

${\displaystyle {\begin{matrix}{}\\{\overline {X}}_{n}\ \xrightarrow {\text{a.s.}} \ \mu \qquad {\textrm {when}}\ n\to \infty .\\{}\end{matrix}}}$ (3)

That is,

${\displaystyle \Pr \!\left(\lim _{n\to \infty }{\overline {X}}_{n}=\mu \right)=1.}$

What this means is that the probability that, as the number of trials n goes to infinity, the average of the observations converges to the expected value, is equal to one. The modern proof of the strong law is more complex than that of the weak law, and relies on passing to an appropriate subsequence.[14]

The strong law of large numbers can itself be seen as a special case of the pointwise ergodic theorem. This view justifies the intuitive interpretation of the expected value (for Lebesgue integration only) of a random variable when sampled repeatedly as the "long-term average".

Law 3 is called the strong law because random variables which converge strongly (almost surely) are guaranteed to converge weakly (in probability). However the weak law is known to hold in certain conditions where the strong law does not hold and then the convergence is only weak (in probability). See #Differences between the weak law and the strong law.

The strong law applies to independent identically distributed random variables having an expected value (like the weak law). This was proved by Kolmogorov in 1930. It can also apply in other cases. Kolmogorov also showed, in 1933, that if the variables are independent and identically distributed, then for the average to converge almost surely on something (this can be considered another statement of the strong law), it is necessary that they have an expected value (and then of course the average will converge almost surely on that).[19]

If the summands are independent but not identically distributed, then

${\displaystyle {\begin{matrix}{}\\{\overline {X}}_{n}-\operatorname {E} {\big [}{\overline {X}}_{n}{\big ]}\ \xrightarrow {\text{a.s.}} \ 0,\\{}\end{matrix}}}$ (2)

provided that each X_k has a finite second moment and

${\displaystyle \sum _{k=1}^{\infty }{\frac {1}{k^{2}}}\operatorname {Var} [X_{k}]<\infty .}$

This statement is known as Kolmogorov's strong law, see e.g. Sen & Singer (1993, Theorem 2.3.10).

The weak law states that for a specified large n, the average ${\displaystyle {\overline {X}}_{n}}$ is likely to be near μ. Thus, it leaves open the possibility that ${\displaystyle |{\overline {X}}_{n}-\mu |>\varepsilon }$ happens an infinite number of times, although at infrequent intervals. (Not necessarily ${\displaystyle |{\overline {X}}_{n}-\mu |\neq 0}$ for all n).

The strong law shows that this almost surely will not occur. Note that it does not imply that with probability 1, we have that for any ε > 0 the inequality ${\displaystyle |{\overline {X}}_{n}-\mu |<\varepsilon }$ holds for all large enough n, since the convergence is not necessarily uniform on the set where it holds.[20]

The strong law does not hold in the following cases, but the weak law does.[21][22]

Suppose f(x,θ) is some function defined for θ ∈ Θ, and continuous in θ. Then for any fixed θ, the sequence {f(X₁,θ), f(X₂,θ), ...} will be a sequence of independent and identically distributed random variables, such that the sample mean of this sequence converges in probability to E[f(X,θ)]. This is the pointwise (in θ) convergence.

The uniform law of large numbers states the conditions under which the convergence happens uniformly in θ. If[25][26]

1. Θ is compact,
2. f(x,θ) is continuous at each θ ∈ Θ for almost all xs, and measurable function of x at each θ.
3. there exists a dominating function d(x) such that E[d(X)] < ∞, and

${\displaystyle \left\|f(x,\theta )\right\|\leq d(x)\quad {\text{for all}}\ \theta \in \Theta .}$

Then E[f(X,θ)] is continuous in θ, and

${\displaystyle \sup _{\theta \in \Theta }\left\|{\frac {1}{n}}\sum _{i=1}^{n}f(X_{i},\theta )-\operatorname {E} [f(X,\theta )]\right\|\xrightarrow {\mathrm {\mathbb {P} } } \ 0.}$

This result is useful to derive consistency of a large class of estimators (see Extremum estimator).

Borel's law of large numbers, named after Émile Borel, states that if an experiment is repeated a large number of times, independently under identical conditions, then the proportion of times that any specified event occurs approximately equals the probability of the event's occurrence on any particular trial; the larger the number of repetitions, the better the approximation tends to be. More precisely, if E denotes the event in question, p its probability of occurrence, and N_n(E) the number of times E occurs in the first n trials, then with probability one,[27]

${\displaystyle {\frac {N_{n}(E)}{n}}\to p{\text{ as }}n\to \infty .}$

This theorem makes rigorous the intuitive notion of probability as the long-run relative frequency of an event's occurrence. It is a special case of any of several more general laws of large numbers in probability theory.

Chebyshev's inequality. Let X be a random variable with finite expected value μ and finite non-zero variance σ². Then for any real number k > 0,

${\displaystyle \Pr(|X-\mu |\geq k\sigma )\leq {\frac {1}{k^{2}}}.}$

This proof uses the assumption of finite variance ${\displaystyle \operatorname {Var} (X_{i})=\sigma ^{2}}$ (for all ${\displaystyle i}$). The independence of the random variables implies no correlation between them, and we have that

$${\displaystyle \operatorname {Var} ({\overline {X}}_{n})=\operatorname {Var} ({\tfrac {1}{n}}(X_{1}+\cdots +X_{n}))={\frac {1}{n^{2}}}\operatorname {Var} (X_{1}+\cdots +X_{n})={\frac {n\sigma ^{2}}{n^{2}}}={\frac {\sigma ^{2}}{n}}.}$$

The common mean μ of the sequence is the mean of the sample average:

$${\displaystyle E({\overline {X}}_{n})=\mu .}$$

Using Chebyshev's inequality on ${\displaystyle {\overline {X}}_{n}}$ results in

$${\displaystyle \operatorname {P} (\left|{\overline {X}}_{n}-\mu \right|\geq \varepsilon )\leq {\frac {\sigma ^{2}}{n\varepsilon ^{2}}}.}$$

This may be used to obtain the following:

$${\displaystyle \operatorname {P} (\left|{\overline {X}}_{n}-\mu \right|<\varepsilon )=1-\operatorname {P} (\left|{\overline {X}}_{n}-\mu \right|\geq \varepsilon )\geq 1-{\frac {\sigma ^{2}}{n\varepsilon ^{2}}}.}$$

As n approaches infinity, the expression approaches 1. And by definition of convergence in probability, we have obtained

${\displaystyle {\begin{matrix}{}\\{\overline {X}}_{n}\ {\xrightarrow {P}}\ \mu \qquad {\textrm {when}}\ n\to \infty .\\{}\end{matrix}}}$ (2)

By Taylor's theorem for complex functions, the characteristic function of any random variable, X, with finite mean μ, can be written as

$${\displaystyle \varphi _{X}(t)=1+it\mu +o(t),\quad t\rightarrow 0.}$$

All X₁, X₂, ... have the same characteristic function, so we will simply denote this φ_X.

Among the basic properties of characteristic functions there are

$${\displaystyle \varphi _{{\frac {1}{n}}X}(t)=\varphi _{X}({\tfrac {t}{n}})\quad {\text{and}}\quad \varphi _{X+Y}(t)=\varphi _{X}(t)\varphi _{Y}(t)\quad }$$

if X and Y are independent.

These rules can be used to calculate the characteristic function of ${\displaystyle \scriptstyle {\overline {X}}_{n}}$ in terms of φ_X:

$${\displaystyle \varphi _{{\overline {X}}_{n}}(t)=\left[\varphi _{X}\left({t \over n}\right)\right]^{n}=\left[1+i\mu {t \over n}+o\left({t \over n}\right)\right]^{n}\,\rightarrow \,e^{it\mu },\quad {\text{as}}\quad n\rightarrow \infty .}$$

The limit  e^itμ  is the characteristic function of the constant random variable μ, and hence by the Lévy continuity theorem, ${\displaystyle \scriptstyle {\overline {X}}_{n}}$ converges in distribution to μ:

$${\displaystyle {\overline {X}}_{n}\,{\xrightarrow {\mathcal {D}}}\,\mu \qquad {\text{for}}\qquad n\to \infty .}$$

μ is a constant, which implies that convergence in distribution to μ and convergence in probability to μ are equivalent (see Convergence of random variables.) Therefore,

${\displaystyle {\begin{matrix}{}\\{\overline {X}}_{n}\ {\xrightarrow {P}}\ \mu \qquad {\textrm {when}}\ n\to \infty .\\{}\end{matrix}}}$ (2)

This shows that the sample mean converges in probability to the derivative of the characteristic function at the origin, as long as the latter exists.