Empirical process

For X₁, X₂, ... X_n independent and identically-distributed random variables in R with common cumulative distribution function F(x), the empirical distribution function is defined by

${\displaystyle F_{n}(x)={\frac {1}{n}}\sum _{i=1}^{n}I_{(-\infty ,x]}(X_{i}),}$

where I_C is the indicator function of the set C.

For every (fixed) x, F_n(x) is a sequence of random variables which converge to F(x) almost surely by the strong law of large numbers. That is, F_n converges to F pointwise. Glivenko and Cantelli strengthened this result by proving uniform convergence of F_n to F by the Glivenko–Cantelli theorem.[5]

A centered and scaled version of the empirical measure is the signed measure

${\displaystyle G_{n}(A)={\sqrt {n}}(P_{n}(A)-P(A))}$

It induces a map on measurable functions f given by

${\displaystyle f\mapsto G_{n}f={\sqrt {n}}(P_{n}-P)f={\sqrt {n}}\left({\frac {1}{n}}\sum _{i=1}^{n}f(X_{i})-\mathbb {E} f\right)}$

By the central limit theorem, ${\displaystyle G_{n}(A)}$ converges in distribution to a normal random variable N(0, P(A)(1 − P(A))) for fixed measurable set A. Similarly, for a fixed function f, ${\displaystyle G_{n}f}$ converges in distribution to a normal random variable ${\displaystyle N(0,\mathbb {E} (f-\mathbb {E} f)^{2})}$, provided that ${\displaystyle \mathbb {E} f}$ and ${\displaystyle \mathbb {E} f^{2}}$ exist.

Definition

${\displaystyle {\bigl (}G_{n}(c){\bigr )}_{c\in {\mathcal {C}}}}$ is called an empirical process indexed by ${\displaystyle {\mathcal {C}}}$, a collection of measurable subsets of S.

${\displaystyle {\bigl (}G_{n}f{\bigr )}_{f\in {\mathcal {F}}}}$ is called an empirical process indexed by ${\displaystyle {\mathcal {F}}}$, a collection of measurable functions from S to ${\displaystyle \mathbb {R} }$.

A significant result in the area of empirical processes is Donsker's theorem. It has led to a study of Donsker classes: sets of functions with the useful property that empirical processes indexed by these classes converge weakly to a certain Gaussian process. While it can be shown that Donsker classes are Glivenko–Cantelli classes, the converse is not true in general.

As an example, consider empirical distribution functions. For real-valued iid random variables X₁, X₂, ..., X_n they are given by

${\displaystyle F_{n}(x)=P_{n}((-\infty ,x])=P_{n}I_{(-\infty ,x]}.}$

In this case, empirical processes are indexed by a class ${\displaystyle {\mathcal {C}}=\{(-\infty ,x]:x\in \mathbb {R} \}.}$ It has been shown that ${\displaystyle {\mathcal {C}}}$ is a Donsker class, in particular,

${\displaystyle {\sqrt {n}}(F_{n}(x)-F(x))}$ converges weakly in ${\displaystyle \ell ^{\infty }(\mathbb {R} )}$ to a Brownian bridge B(F(x)) .

• Khmaladze transformation
• Weak convergence of measures
• Glivenko–Cantelli theorem

• Billingsley, P. (1995). Probability and Measure (Third ed.). New York: John Wiley and Sons. ISBN 0471007102.
• Donsker, M. D. (1952). "Justification and Extension of Doob's Heuristic Approach to the Kolmogorov- Smirnov Theorems". The Annals of Mathematical Statistics. 23 (2): 277–281. doi:10.1214/aoms/1177729445.
• Dudley, R. M. (1978). "Central Limit Theorems for Empirical Measures". The Annals of Probability. 6 (6): 899–929. doi:10.1214/aop/1176995384.
• Dudley, R. M. (1999). Uniform Central Limit Theorems. Cambridge Studies in Advanced Mathematics. Vol. 63. Cambridge, UK: Cambridge University Press.
• Kosorok, M. R. (2008). Introduction to Empirical Processes and Semiparametric Inference. Springer Series in Statistics. doi:10.1007/978-0-387-74978-5. ISBN 978-0-387-74977-8.
• Shorack, G. R.; Wellner, J. A. (2009). Empirical Processes with Applications to Statistics. doi:10.1137/1.9780898719017. ISBN 978-0-89871-684-9.
• van der Vaart, Aad W.; Wellner, Jon A. (2000). Weak Convergence and Empirical Processes: With Applications to Statistics (2nd ed.). Springer. ISBN 978-0-387-94640-5.
• Dzhaparidze, K. O.; Nikulin, M. S. (1982). "Probability distributions of the Kolmogorov and omega-square statistics for continuous distributions with shift and scale parameters". Journal of Soviet Mathematics. 20 (3): 2147. doi:10.1007/BF01239992. S2CID 123206522.

• Empirical Processes: Theory and Applications, by David Pollard, a textbook available online.
• Introduction to Empirical Processes and Semiparametric Inference, by Michael Kosorok, another textbook available online.