Linear prediction

The most common choice in optimization of parameters ${\displaystyle a_{i}}$ is the root mean square criterion which is also called the autocorrelation criterion. In this method we minimize the expected value of the squared error ${\displaystyle E[e^{2}(n)]}$, which yields the equation

${\displaystyle \sum _{i=1}^{p}a_{i}R(j-i)=R(j),}$

for 1 ≤ j ≤ p, where R is the autocorrelation of signal x_n, defined as

${\displaystyle \ R(i)=E\{x(n)x(n-i)\}\,}$,

and E is the expected value. In the multi-dimensional case this corresponds to minimizing the L₂ norm.

The above equations are called the normal equations or Yule-Walker equations. In matrix form the equations can be equivalently written as

${\displaystyle \mathbf {RA} =\mathbf {r} }$

where the autocorrelation matrix ${\displaystyle \mathbf {R} }$ is a symmetric, ${\displaystyle p\times p}$ Toeplitz matrix with elements ${\displaystyle r_{ij}=R(i-j),0\leq i,j<p}$, the vector ${\displaystyle \mathbf {r} }$ is the autocorrelation vector ${\displaystyle r_{j}=R(j),0<j\leq p}$, and ${\displaystyle \mathbf {A} =[a_{1},a_{2},\,\cdots \,,a_{p-1},a_{p}]}$, the parameter vector.

Another, more general, approach is to minimize the sum of squares of the errors defined in the form

${\displaystyle e(n)=x(n)-{\widehat {x}}(n)=x(n)-\sum _{i=1}^{p}a_{i}x(n-i)=-\sum _{i=0}^{p}a_{i}x(n-i)}$

where the optimisation problem searching over all ${\displaystyle a_{i}}$ must now be constrained with ${\displaystyle a_{0}=-1}$.

On the other hand, if the mean square prediction error is constrained to be unity and the prediction error equation is included on top of the normal equations, the augmented set of equations is obtained as

${\displaystyle \ \mathbf {RA} =[1,0,...,0]^{\mathrm {T} }}$

where the index ${\displaystyle i}$ ranges from 0 to ${\displaystyle p}$, and ${\displaystyle \mathbf {R} }$ is a ${\displaystyle (p+1)\times (p+1)}$ matrix.

Specification of the parameters of the linear predictor is a wide topic and a large number of other approaches have been proposed. In fact, the autocorrelation method is the most common[2] and it is used, for example, for speech coding in the GSM standard.

Solution of the matrix equation ${\displaystyle \mathbf {RA} =\mathbf {r} }$ is computationally a relatively expensive process. The Gaussian elimination for matrix inversion is probably the oldest solution but this approach does not efficiently use the symmetry of ${\displaystyle \mathbf {R} }$. A faster algorithm is the Levinson recursion proposed by Norman Levinson in 1947, which recursively calculates the solution. In particular, the autocorrelation equations above may be more efficiently solved by the Durbin algorithm.[3]

In 1986, Philippe Delsarte and Y.V. Genin proposed an improvement to this algorithm called the split Levinson recursion, which requires about half the number of multiplications and divisions.[4] It uses a special symmetrical property of parameter vectors on subsequent recursion levels. That is, calculations for the optimal predictor containing ${\displaystyle p}$ terms make use of similar calculations for the optimal predictor containing ${\displaystyle p-1}$ terms.

Another way of identifying model parameters is to iteratively calculate state estimates using Kalman filters and obtaining maximum likelihood estimates within expectation–maximization algorithms.

For equally-spaced values, a polynomial interpolation is a linear combination of the known values. If the discrete time signal is estimated to obey a polynomial of degree ${\displaystyle p-1,}$ then the predictor coefficients ${\displaystyle a_{i}}$ are given by the corresponding row of the triangle of binomial transform coefficients. This estimate might be suitable for a slowly varying signal with low noise. The predictions for the first few values of ${\displaystyle p}$ are

${\displaystyle {\begin{array}{lcl}p=1&:&{\widehat {x}}(n)=1x(n-1)\\p=2&:&{\widehat {x}}(n)=2x(n-1)-1x(n-2)\\p=3&:&{\widehat {x}}(n)=3x(n-1)-3x(n-2)+1x(n-3)\\p=4&:&{\widehat {x}}(n)=4x(n-1)-6x(n-2)+4x(n-3)-1x(n-4)\\\end{array}}}$