Rate of convergence

Suppose that the sequence ${\displaystyle (x_{k})}$ converges to the number ${\displaystyle L}$. The sequence is said to converge Q-linearly to ${\displaystyle L}$ if there exists a number ${\displaystyle \mu \in (0,1)}$ such that

${\displaystyle \lim _{k\to \infty }{\frac {|x_{k+1}-L|}{|x_{k}-L|}}=\mu .}$

The number ${\displaystyle \mu }$ is called the rate of convergence.[3]

The sequence is said to converge Q-superlinearly to ${\displaystyle L}$ (i.e. faster than linearly) if

${\displaystyle \lim _{k\to \infty }{\frac {|x_{k+1}-L|}{|x_{k}-L|}}=0}$

and it is said to converge Q-sublinearly to ${\displaystyle L}$ (i.e. slower than linearly) if

${\displaystyle \lim _{k\to \infty }{\frac {|x_{k+1}-L|}{|x_{k}-L|}}=1.}$

If the sequence converges sublinearly and additionally

${\displaystyle \lim _{k\to \infty }{\frac {|x_{k+2}-x_{k+1}|}{|x_{k+1}-x_{k}|}}=1,}$

then it is said that the sequence ${\displaystyle (x_{k})}$ converges logarithmically to ${\displaystyle L}$.[4] Note that unlike previous definitions, logarithmic convergence is not called "Q-logarithmic."

In order to further classify convergence, the order of convergence is defined as follows. The sequence is said to converge with order ${\displaystyle q}$ to ${\displaystyle L}$ for ${\displaystyle q\geq 1}$ if

${\displaystyle \lim _{k\to \infty }{\frac {|x_{k+1}-L|}{|x_{k}-L|^{q}}}<M}$

for some positive constant ${\displaystyle M>0}$ (not necessarily less than 1 if ${\displaystyle q>1}$). In particular, convergence with order

• ${\displaystyle q=1}$ is called linear convergence (if ${\displaystyle M<1}$),
• ${\displaystyle q=2}$ is called quadratic convergence,
• ${\displaystyle q=3}$ is called cubic convergence,
• etc.

Some sources require that ${\displaystyle q}$ is strictly greater than ${\displaystyle 1}$ since the ${\displaystyle q=1}$ case requires ${\displaystyle M<1}$ so is best treated separately.[5] It is not necessary, however, that ${\displaystyle q}$ be an integer. For example, the secant method, when converging to a regular, simple root, has an order of φ ≈ 1.618.

In the definitions above, the "Q-" stands for "quotient" because the terms are defined using the quotient between two successive terms.[6]^: 619 Often, however, the "Q-" is dropped and a sequence is simply said to have linear convergence, quadratic convergence, etc.

A practical method to calculate the order of convergence for a sequence is to calculate the following sequence, which converges to ${\displaystyle q}$

${\displaystyle q\approx {\frac {\log \left|{\frac {x_{k+1}-x_{k}}{x_{k}-x_{k-1}}}\right|}{\log \left|{\frac {x_{k}-x_{k-1}}{x_{k-1}-x_{k-2}}}\right|}}.}$[7]

The Q-convergence definitions have a shortcoming in that they do not include some sequences, such as the sequence ${\displaystyle (b_{k})}$ below, which converge reasonably fast, but whose rate is variable. Therefore, the definition of rate of convergence is extended as follows.

Suppose that ${\displaystyle (x_{k})}$ converges to ${\displaystyle L}$. The sequence is said to converge R-linearly to ${\displaystyle L}$ if there exists a sequence ${\displaystyle (\varepsilon _{k})}$ such that

${\displaystyle |x_{k}-L|\leq \varepsilon _{k}\quad {\text{for all }}k\,,}$

and ${\displaystyle (\varepsilon _{k})}$ converges Q-linearly to zero.[3] The "R-" prefix stands for "root". [6]^: 620

Consider the sequence

${\displaystyle (a_{k})=\left\{1,{\frac {1}{2}},{\frac {1}{4}},{\frac {1}{8}},{\frac {1}{16}},{\frac {1}{32}},\ldots ,{\frac {1}{2^{k}}},...\right\}.}$

It can be shown that this sequence converges to ${\displaystyle L=0}$. To determine the type of convergence, we plug the sequence into the definition of Q-linear convergence,

${\displaystyle \lim _{k\to \infty }{\frac {\left|1/2^{k+1}-0\right|}{\left|1/2^{k}-0\right|}}=\lim _{k\to \infty }{\frac {2^{k}}{2^{k+1}}}={\frac {1}{2}}.}$

Thus, we find that ${\displaystyle (a_{k})}$ converges Q-linearly and has a convergence rate of ${\displaystyle \mu =1/2}$. More generally, for any ${\displaystyle c\in \mathbb {R} ,\mu \in (-1,1)}$, the sequence ${\displaystyle (c\mu ^{k})}$ converges linearly with rate ${\displaystyle |\mu |}$.

The sequence

${\displaystyle (b_{k})=\left\{1,1,{\frac {1}{4}},{\frac {1}{4}},{\frac {1}{16}},{\frac {1}{16}},\ldots ,{\frac {1}{4^{\left\lfloor {\frac {k}{2}}\right\rfloor }}},\,\ldots \right\}}$

also converges linearly to 0 with rate 1/2 under the R-convergence definition, but not under the Q-convergence definition. (Note that ${\displaystyle \lfloor x\rfloor }$ is the floor function, which gives the largest integer that is less than or equal to ${\displaystyle x}$.)

The sequence

${\displaystyle (c_{k})=\left\{{\frac {1}{2}},{\frac {1}{4}},{\frac {1}{16}},{\frac {1}{256}},{\frac {1}{65,\!536}},\ldots ,{\frac {1}{2^{2^{k}}}},\ldots \right\}}$

converges superlinearly. In fact, it is quadratically convergent.

Finally, the sequence

${\displaystyle (d_{k})=\left\{1,{\frac {1}{2}},{\frac {1}{3}},{\frac {1}{4}},{\frac {1}{5}},{\frac {1}{6}},\ldots ,{\frac {1}{k+1}},\ldots \right\}}$

converges sublinearly and logarithmically.

Linear, linear, superlinear (quadratic), and sublinear rates of convergence

Consider the ordinary differential equation

${\displaystyle {\frac {dy}{dx}}=-\kappa y}$

with initial condition ${\displaystyle y(0)=y_{0}}$. We can solve this equation using the Forward Euler scheme for numerical discretization:

${\displaystyle {\frac {y_{n+1}-y_{n}}{h}}=-\kappa y_{n},}$

which generates the sequence

${\displaystyle y_{n+1}=y_{n}(1-h\kappa ).}$

In terms of ${\displaystyle y(0)=y_{0}}$, this sequence is as follows, from the Binomial theorem:

${\displaystyle y_{n}=y_{0}(1-h\kappa )^{n}=y_{0}\left(1-nh\kappa +n(n-1){\frac {h^{2}\kappa ^{2}}{2}}+....\right).}$

The exact solution to this ODE is ${\displaystyle y=f(x)=y_{0}\exp(-\kappa x)}$, corresponding to the following Taylor expansion in ${\displaystyle h\kappa }$ for ${\displaystyle h\kappa \ll 1}$:

${\displaystyle f(x_{n})=f(nh)=y_{0}\exp(-\kappa nh)=y_{0}\left[\exp(-\kappa h)\right]^{n}=y_{0}\left(1-h\kappa +{\frac {h^{2}\kappa ^{2}}{2}}+....\right)^{n}=y_{0}\left(1-nh\kappa +{\frac {n^{2}h^{2}\kappa ^{2}}{2}}+...\right).}$

In this case, the truncation error is

${\displaystyle e=|y_{n}-f(x_{n})|={\frac {nh^{2}\kappa ^{2}}{2}}={\mathcal {O}}(h^{2}),}$

so ${\displaystyle (y_{n})}$ converges to ${\displaystyle f(x_{n})}$ with a convergence rate ${\displaystyle q=2}$.

The sequence ${\displaystyle (d_{k})}$ with ${\displaystyle d_{k}=1/(k+1)}$ was introduced above. This sequence converges with order 1 according to the convention for discretization methods.

The sequence ${\displaystyle (a_{k})}$ with ${\displaystyle a_{k}=2^{-k}}$, which was also introduced above, converges with order q for every number q. It is said to converge exponentially using the convention for discretization methods. However, it only converges linearly (that is, with order 1) using the convention for iterative methods.