Whitening transformation

Suppose ${\displaystyle X}$ is a random (column) vector with non-singular covariance matrix ${\displaystyle \Sigma }$ and mean ${\displaystyle 0}$. Then the transformation ${\displaystyle Y=WX}$ with a whitening matrix ${\displaystyle W}$ satisfying the condition ${\displaystyle W^{\mathrm {T} }W=\Sigma ^{-1}}$ yields the whitened random vector ${\displaystyle Y}$ with unit diagonal covariance.

There are infinitely many possible whitening matrices ${\displaystyle W}$ that all satisfy the above condition. Commonly used choices are ${\displaystyle W=\Sigma ^{-1/2}}$ (Mahalanobis or ZCA whitening), ${\displaystyle W=L^{T}}$ where ${\displaystyle L}$ is the Cholesky decomposition of ${\displaystyle \Sigma ^{-1}}$ (Cholesky whitening),[3] or the eigen-system of ${\displaystyle \Sigma }$ (PCA whitening).[4]

Optimal whitening transforms can be singled out by investigating the cross-covariance and cross-correlation of ${\displaystyle X}$ and ${\displaystyle Y}$.[3] For example, the unique optimal whitening transformation achieving maximal component-wise correlation between original ${\displaystyle X}$ and whitened ${\displaystyle Y}$ is produced by the whitening matrix ${\displaystyle W=P^{-1/2}V^{-1/2}}$ where ${\displaystyle P}$ is the correlation matrix and ${\displaystyle V}$ the variance matrix.

Whitening a data matrix follows the same transformation as for random variables. An empirical whitening transform is obtained by estimating the covariance (e.g. by maximum likelihood) and subsequently constructing a corresponding estimated whitening matrix (e.g. by Cholesky decomposition).

This modality is a generalization of the pre-whitening procedure extended to more general spaces where ${\displaystyle X}$ is usually assumed to be a random function or other random objects in a Hilbert space ${\displaystyle H}$. One of the main issues of extending whitening to infinite dimensions is that the covariance operator has an unbounded inverse in ${\displaystyle H}$. Nevertheless, if one assumes that Picard condition holds for ${\displaystyle X}$ in the range space of the covariance operator, whitening becomes possible.[5] A whitening operator can be then defined from the factorization of the Moore–Penrose inverse of the covariance operator, which has effective mapping on Karhunen–Loève type expansions of ${\displaystyle X}$. The advantage of these whitening transformations is that they can be optimized according to the underlying topological properties of the data (smoothness, continuity and contiguity), thus producing more robust whitening representations. High-dimensional features of the data can be exploited through kernel regressors or basis function systems.[6]

An implementation of several whitening procedures in R, including ZCA-whitening and PCA whitening but also CCA whitening, is available in the "whitening" R package [7] published on CRAN. The R package "pfica"[8] allows the computation of high-dimensional whitening representations using basis function systems (B-splines, Fourier basis, etc.).

• Decorrelation
• Principal component analysis
• Weighted least squares
• Canonical correlation
• Mahalanobis distance (is Euclidean after W. transformation).

• http://courses.media.mit.edu/2010fall/mas622j/whiten.pdf
• The ZCA whitening transformation. Appendix A of Learning Multiple Layers of Features from Tiny Images by A. Krizhevsky.