Rayleigh quotient

As stated in the introduction, for any vector x, one has ${\displaystyle R(M,x)\in \left[\lambda _{\min },\lambda _{\max }\right]}$, where ${\displaystyle \lambda _{\min },\lambda _{\max }}$ are respectively the smallest and largest eigenvalues of ${\displaystyle M}$. This is immediate after observing that the Rayleigh quotient is a weighted average of eigenvalues of M:

$${\displaystyle R(M,x)={x^{*}Mx \over x^{*}x}={\frac {\sum _{i=1}^{n}\lambda _{i}y_{i}^{2}}{\sum _{i=1}^{n}y_{i}^{2}}}}$$

where ${\displaystyle (\lambda _{i},v_{i})}$ is the ${\displaystyle i}$-th eigenpair after orthonormalization and ${\displaystyle y_{i}=v_{i}^{*}x}$ is the ${\displaystyle i}$th coordinate of x in the eigenbasis. It is then easy to verify that the bounds are attained at the corresponding eigenvectors ${\displaystyle v_{\min },v_{\max }}$.

The fact that the quotient is a weighted average of the eigenvalues can be used to identify the second, the third, ... largest eigenvalues. Let ${\displaystyle \lambda _{\max }=\lambda _{1}\geq \lambda _{2}\geq \cdots \geq \lambda _{n}=\lambda _{\min }}$ be the eigenvalues in decreasing order. If ${\displaystyle n=2}$ and ${\displaystyle x}$ is constrained to be orthogonal to ${\displaystyle v_{1}}$, in which case ${\displaystyle y_{1}=v_{1}^{*}x=0}$, then ${\displaystyle R(M,x)}$ has maximum value ${\displaystyle \lambda _{2}}$, which is achieved when ${\displaystyle x=v_{2}}$.

An empirical covariance matrix ${\displaystyle M}$ can be represented as the product ${\displaystyle A'A}$ of the data matrix ${\displaystyle A}$ pre-multiplied by its transpose ${\displaystyle A'}$. Being a positive semi-definite matrix, ${\displaystyle M}$ has non-negative eigenvalues, and orthogonal (or orthogonalisable) eigenvectors, which can be demonstrated as follows.

Firstly, that the eigenvalues ${\displaystyle \lambda _{i}}$ are non-negative:

$${\displaystyle {\begin{aligned}&Mv_{i}=A'Av_{i}=\lambda _{i}v_{i}\\\Rightarrow {}&v_{i}'A'Av_{i}=v_{i}'\lambda _{i}v_{i}\\\Rightarrow {}&\left\|Av_{i}\right\|^{2}=\lambda _{i}\left\|v_{i}\right\|^{2}\\\Rightarrow {}&\lambda _{i}={\frac {\left\|Av_{i}\right\|^{2}}{\left\|v_{i}\right\|^{2}}}\geq 0.\end{aligned}}}$$

Secondly, that the eigenvectors ${\displaystyle v_{i}}$ are orthogonal to one another:

$${\displaystyle {\begin{aligned}&Mv_{i}=\lambda _{i}v_{i}\\\Rightarrow {}&v_{j}'Mv_{i}=v_{j}'\lambda _{i}v_{i}\\\Rightarrow {}&\left(Mv_{j}\right)'v_{i}=\lambda _{i}v_{j}'v_{i}\\\Rightarrow {}&\lambda _{j}v_{j}'v_{i}=\lambda _{i}v_{j}'v_{i}\\\Rightarrow {}&\left(\lambda _{j}-\lambda _{i}\right)v_{j}'v_{i}=0\\\Rightarrow {}&v_{j}'v_{i}=0\end{aligned}}}$$

if the eigenvalues are different – in the case of multiplicity, the basis can be orthogonalized.

To now establish that the Rayleigh quotient is maximized by the eigenvector with the largest eigenvalue, consider decomposing an arbitrary vector ${\displaystyle x}$ on the basis of the eigenvectors ${\displaystyle v_{i}}$:

$${\displaystyle x=\sum _{i=1}^{n}\alpha _{i}v_{i},}$$

where

$${\displaystyle \alpha _{i}={\frac {x'v_{i}}{v_{i}'v_{i}}}={\frac {\langle x,v_{i}\rangle }{\left\|v_{i}\right\|^{2}}}}$$

is the coordinate of ${\displaystyle x}$ orthogonally projected onto ${\displaystyle v_{i}}$. Therefore, we have:

$${\displaystyle {\begin{aligned}R(M,x)&={\frac {x'A'Ax}{x'x}}\\&={\frac {{\Bigl (}\sum _{j=1}^{n}\alpha _{j}v_{j}{\Bigr )}'\left(A'A\right){\Bigl (}\sum _{i=1}^{n}\alpha _{i}v_{i}{\Bigr )}}{{\Bigl (}\sum _{j=1}^{n}\alpha _{j}v_{j}{\Bigr )}'{\Bigl (}\sum _{i=1}^{n}\alpha _{i}v_{i}{\Bigr )}}}\\&={\frac {{\Bigl (}\sum _{j=1}^{n}\alpha _{j}v_{j}{\Bigr )}'{\Bigl (}\sum _{i=1}^{n}\alpha _{i}(A'A)v_{i}{\Bigr )}}{{\Bigl (}\sum _{i=1}^{n}\alpha _{i}^{2}{v_{i}}'{v_{i}}{\Bigr )}}}\\&={\frac {{\Bigl (}\sum _{j=1}^{n}\alpha _{j}v_{j}{\Bigr )}'{\Bigl (}\sum _{i=1}^{n}\alpha _{i}\lambda _{i}v_{i}{\Bigr )}}{{\Bigl (}\sum _{i=1}^{n}\alpha _{i}^{2}\|{v_{i}}\|^{2}{\Bigr )}}}\end{aligned}}}$$

which, by orthonormality of the eigenvectors, becomes:

$${\displaystyle {\begin{aligned}R(M,x)&={\frac {\sum _{i=1}^{n}\alpha _{i}^{2}\lambda _{i}}{\sum _{i=1}^{n}\alpha _{i}^{2}}}\\&=\sum _{i=1}^{n}\lambda _{i}{\frac {(x'v_{i})^{2}}{(x'x)(v_{i}'v_{i})^{2}}}\\&=\sum _{i=1}^{n}\lambda _{i}{\frac {(x'v_{i})^{2}}{(x'x)}}\end{aligned}}}$$

The last representation establishes that the Rayleigh quotient is the sum of the squared cosines of the angles formed by the vector ${\displaystyle x}$ and each eigenvector ${\displaystyle v_{i}}$, weighted by corresponding eigenvalues.

If a vector ${\displaystyle x}$ maximizes ${\displaystyle R(M,x)}$, then any non-zero scalar multiple ${\displaystyle kx}$ also maximizes ${\displaystyle R}$, so the problem can be reduced to the Lagrange problem of maximizing ${\textstyle \sum _{i=1}^{n}\alpha _{i}^{2}\lambda _{i}}$ under the constraint that ${\textstyle \sum _{i=1}^{n}\alpha _{i}^{2}=1}$.

Define: ${\displaystyle \beta _{i}=\alpha _{i}^{2}}$. This then becomes a linear program, which always attains its maximum at one of the corners of the domain. A maximum point will have ${\displaystyle \alpha _{1}=\pm 1}$ and ${\displaystyle \alpha _{i}=0}$ for all ${\displaystyle i>1}$ (when the eigenvalues are ordered by decreasing magnitude).

Thus, the Rayleigh quotient is maximized by the eigenvector with the largest eigenvalue.

Formulation using Lagrange multipliers

Alternatively, this result can be arrived at by the method of Lagrange multipliers. The first part is to show that the quotient is constant under scaling ${\displaystyle x\to cx}$, where ${\displaystyle c}$ is a scalar

$${\displaystyle R(M,cx)={\frac {(cx)^{*}Mcx}{(cx)^{*}cx}}={\frac {c^{*}c}{c^{*}c}}{\frac {x^{*}Mx}{x^{*}x}}=R(M,x).}$$

Because of this invariance, it is sufficient to study the special case ${\displaystyle \|x\|^{2}=x^{T}x=1}$. The problem is then to find the critical points of the function

$${\displaystyle R(M,x)=x^{\mathsf {T}}Mx,}$$

subject to the constraint ${\displaystyle \|x\|^{2}=x^{T}x=1.}$ In other words, it is to find the critical points of

$${\displaystyle {\mathcal {L}}(x)=x^{\mathsf {T}}Mx-\lambda \left(x^{\mathsf {T}}x-1\right),}$$

where ${\displaystyle \lambda }$ is a Lagrange multiplier. The stationary points of ${\displaystyle {\mathcal {L}}(x)}$ occur at

$${\displaystyle {\begin{aligned}&{\frac {d{\mathcal {L}}(x)}{dx}}=0\\\Rightarrow {}&2x^{\mathsf {T}}M-2\lambda x^{\mathsf {T}}=0\\\Rightarrow {}&2Mx-2\lambda x=0{\text{ (taking the transpose of both sides and noting that }}M{\text{ is Hermitian)}}\\\Rightarrow {}&Mx=\lambda x\end{aligned}}}$$

and

$${\displaystyle \therefore R(M,x)={\frac {x^{\mathsf {T}}Mx}{x^{\mathsf {T}}x}}=\lambda {\frac {x^{\mathsf {T}}x}{x^{\mathsf {T}}x}}=\lambda .}$$

Therefore, the eigenvectors ${\displaystyle x_{1},\ldots ,x_{n}}$ of ${\displaystyle M}$ are the critical points of the Rayleigh quotient and their corresponding eigenvalues ${\displaystyle \lambda _{1},\ldots ,\lambda _{n}}$ are the stationary values of ${\displaystyle {\mathcal {L}}}$. This property is the basis for principal components analysis and canonical correlation.

Sturm–Liouville theory concerns the action of the linear operator

$${\displaystyle L(y)={\frac {1}{w(x)}}\left(-{\frac {d}{dx}}\left[p(x){\frac {dy}{dx}}\right]+q(x)y\right)}$$

on the inner product space defined by

$${\displaystyle \langle {y_{1},y_{2}}\rangle =\int _{a}^{b}w(x)y_{1}(x)y_{2}(x)\,dx}$$

of functions satisfying some specified boundary conditions at a and b. In this case the Rayleigh quotient is

$${\displaystyle {\frac {\langle {y,Ly}\rangle }{\langle {y,y}\rangle }}={\frac {\int _{a}^{b}y(x)\left(-{\frac {d}{dx}}\left[p(x){\frac {dy}{dx}}\right]+q(x)y(x)\right)dx}{\int _{a}^{b}{w(x)y(x)^{2}}dx}}.}$$

This is sometimes presented in an equivalent form, obtained by separating the integral in the numerator and using integration by parts:

$${\displaystyle {\begin{aligned}{\frac {\langle {y,Ly}\rangle }{\langle {y,y}\rangle }}&={\frac {\left\{\int _{a}^{b}y(x)\left(-{\frac {d}{dx}}\left[p(x)y'(x)\right]\right)dx\right\}+\left\{\int _{a}^{b}{q(x)y(x)^{2}}\,dx\right\}}{\int _{a}^{b}{w(x)y(x)^{2}}\,dx}}\\&={\frac {\left\{\left.-y(x)\left[p(x)y'(x)\right]\right|_{a}^{b}\right\}+\left\{\int _{a}^{b}y'(x)\left[p(x)y'(x)\right]\,dx\right\}+\left\{\int _{a}^{b}{q(x)y(x)^{2}}\,dx\right\}}{\int _{a}^{b}w(x)y(x)^{2}\,dx}}\\&={\frac {\left\{\left.-p(x)y(x)y'(x)\right|_{a}^{b}\right\}+\left\{\int _{a}^{b}\left[p(x)y'(x)^{2}+q(x)y(x)^{2}\right]\,dx\right\}}{\int _{a}^{b}{w(x)y(x)^{2}}\,dx}}.\end{aligned}}}$$

1. For a given pair (A, B) of matrices, and a given non-zero vector x, the generalized Rayleigh quotient is defined as:
   $${\displaystyle R(A,B;x):={\frac {x^{*}Ax}{x^{*}Bx}}.}$$

   
   The Generalized Rayleigh Quotient can be reduced to the Rayleigh Quotient ${\displaystyle R(D,C^{*}x)}$ through the transformation ${\displaystyle D=C^{-1}A{C^{*}}^{-1}}$ where ${\displaystyle CC^{*}}$ is the Cholesky decomposition of the Hermitian positive-definite matrix B.
2. For a given pair (x, y) of non-zero vectors, and a given Hermitian matrix H, the generalized Rayleigh quotient can be defined as:
   $${\displaystyle R(H;x,y):={\frac {y^{*}Hx}{\sqrt {y^{*}y\cdot x^{*}x}}}}$$

   
   which coincides with R(H,x) when x = y. In quantum mechanics, this quantity is called a "matrix element" or sometimes a "transition amplitude".

• Field of values
• Min-max theorem
• Rayleigh's quotient in vibrations analysis
• Dirichlet eigenvalue

• Shi Yu, Léon-Charles Tranchevent, Bart Moor, Yves Moreau, Kernel-based Data Fusion for Machine Learning: Methods and Applications in Bioinformatics and Text Mining, Ch. 2, Springer, 2011.