Linear least squares

The three main linear least squares formulations are:

• Ordinary least squares (OLS) is the most common estimator. OLS estimates are commonly used to analyze both experimental and observational data.
  The OLS method minimizes the sum of squared residuals, and leads to a closed-form expression for the estimated value of the unknown parameter vector β:
  $${\displaystyle {\hat {\boldsymbol {\beta }}}=(\mathbf {X} ^{\mathsf {T}}\mathbf {X} )^{-1}\mathbf {X} ^{\mathsf {T}}\mathbf {y} ,}$$

  
  where ${\displaystyle \mathbf {y} }$ is a vector whose ith element is the ith observation of the dependent variable, and ${\displaystyle \mathbf {X} }$ is a matrix whose ij element is the ith observation of the jth independent variable. The estimator is unbiased and consistent if the errors have finite variance and are uncorrelated with the regressors:[1]
  $${\displaystyle \operatorname {E} [\,\mathbf {x} _{i}\varepsilon _{i}\,]=0,}$$

  
  where ${\displaystyle \mathbf {x} _{i}}$ is the transpose of row i of the matrix ${\displaystyle \mathbf {X} .}$ It is also efficient under the assumption that the errors have finite variance and are homoscedastic, meaning that E[ε_i²|x_i] does not depend on i. The condition that the errors are uncorrelated with the regressors will generally be satisfied in an experiment, but in the case of observational data, it is difficult to exclude the possibility of an omitted covariate z that is related to both the observed covariates and the response variable. The existence of such a covariate will generally lead to a correlation between the regressors and the response variable, and hence to an inconsistent estimator of β. The condition of homoscedasticity can fail with either experimental or observational data. If the goal is either inference or predictive modeling, the performance of OLS estimates can be poor if multicollinearity is present, unless the sample size is large.
• Weighted least squares (WLS) are used when heteroscedasticity is present in the error terms of the model.
• Generalized least squares (GLS) is an extension of the OLS method, that allows efficient estimation of β when either heteroscedasticity, or correlations, or both are present among the error terms of the model, as long as the form of heteroscedasticity and correlation is known independently of the data. To handle heteroscedasticity when the error terms are uncorrelated with each other, GLS minimizes a weighted analogue to the sum of squared residuals from OLS regression, where the weight for the i^th case is inversely proportional to var(ε_i). This special case of GLS is called "weighted least squares". The GLS solution to an estimation problem is
  $${\displaystyle {\hat {\boldsymbol {\beta }}}=(\mathbf {X} ^{\mathsf {T}}{\boldsymbol {\Omega }}^{-1}\mathbf {X} )^{-1}\mathbf {X} ^{\mathsf {T}}{\boldsymbol {\Omega }}^{-1}\mathbf {y} ,}$$

  
  where Ω is the covariance matrix of the errors. GLS can be viewed as applying a linear transformation to the data so that the assumptions of OLS are met for the transformed data. For GLS to be applied, the covariance structure of the errors must be known up to a multiplicative constant.

Other formulations include:

• Iteratively reweighted least squares (IRLS) is used when heteroscedasticity, or correlations, or both are present among the error terms of the model, but where little is known about the covariance structure of the errors independently of the data.[2] In the first iteration, OLS, or GLS with a provisional covariance structure is carried out, and the residuals are obtained from the fit. Based on the residuals, an improved estimate of the covariance structure of the errors can usually be obtained. A subsequent GLS iteration is then performed using this estimate of the error structure to define the weights. The process can be iterated to convergence, but in many cases, only one iteration is sufficient to achieve an efficient estimate of β.[3][4]
• Instrumental variables regression (IV) can be performed when the regressors are correlated with the errors. In this case, we need the existence of some auxiliary instrumental variables z_i such that E[z_iε_i] = 0. If Z is the matrix of instruments, then the estimator can be given in closed form as
  $${\displaystyle {\hat {\boldsymbol {\beta }}}=(\mathbf {X} ^{\mathsf {T}}\mathbf {Z} (\mathbf {Z} ^{\mathsf {T}}\mathbf {Z} )^{-1}\mathbf {Z} ^{\mathsf {T}}\mathbf {X} )^{-1}\mathbf {X} ^{\mathsf {T}}\mathbf {Z} (\mathbf {Z} ^{\mathsf {T}}\mathbf {Z} )^{-1}\mathbf {Z} ^{\mathsf {T}}\mathbf {y} .}$$

  
  Optimal instruments regression is an extension of classical IV regression to the situation where E[ε_i | z_i] = 0.
• Total least squares (TLS)[5] is an approach to least squares estimation of the linear regression model that treats the covariates and response variable in a more geometrically symmetric manner than OLS. It is one approach to handling the "errors in variables" problem, and is also sometimes used even when the covariates are assumed to be error-free.

• Percentage least squares focuses on reducing percentage errors, which is useful in the field of forecasting or time series analysis. It is also useful in situations where the dependent variable has a wide range without constant variance, as here the larger residuals at the upper end of the range would dominate if OLS were used. When the percentage or relative error is normally distributed, least squares percentage regression provides maximum likelihood estimates. Percentage regression is linked to a multiplicative error model, whereas OLS is linked to models containing an additive error term.[6]

• Constrained least squares, indicates a linear least squares problem with additional constraints on the solution.

In OLS (i.e., assuming unweighted observations), the optimal value of the objective function is found by substituting the optimal expression for the coefficient vector:

$${\displaystyle S=\mathbf {y} ^{\mathsf {T}}(\mathbf {I} -\mathbf {H} )^{\mathsf {T}}(\mathbf {I} -\mathbf {H} )\mathbf {y} =\mathbf {y} ^{\mathsf {T}}(\mathbf {I} -\mathbf {H} )\mathbf {y} ,}$$

where ${\displaystyle \mathbf {H} =\mathbf {X} (\mathbf {X} ^{\mathsf {T}}\mathbf {X} )^{-1}\mathbf {X} ^{\mathsf {T}}}$, the latter equality holding since ${\displaystyle (\mathbf {I} -\mathbf {H} )}$ is symmetric and idempotent. It can be shown from this[7] that under an appropriate assignment of weights the expected value of S is m − n. If instead unit weights are assumed, the expected value of S is ${\displaystyle (m-n)\sigma ^{2}}$, where ${\displaystyle \sigma ^{2}}$ is the variance of each observation.

If it is assumed that the residuals belong to a normal distribution, the objective function, being a sum of weighted squared residuals, will belong to a chi-squared (${\displaystyle \chi ^{2}}$) distribution with m − n degrees of freedom. Some illustrative percentile values of ${\displaystyle \chi ^{2}}$ are given in the following table.[8]

${\displaystyle m-n}$	${\displaystyle \chi _{0.50}^{2}}$	${\displaystyle \chi _{0.95}^{2}}$	${\displaystyle \chi _{0.99}^{2}}$
10	9.34	18.3	23.2
25	24.3	37.7	44.3
100	99.3	124	136

These values can be used for a statistical criterion as to the goodness of fit. When unit weights are used, the numbers should be divided by the variance of an observation.

For WLS, the ordinary objective function above is replaced for a weighted average of residuals.

In statistics and mathematics, linear least squares is an approach to fitting a mathematical or statistical model to data in cases where the idealized value provided by the model for any data point is expressed linearly in terms of the unknown parameters of the model. The resulting fitted model can be used to summarize the data, to predict unobserved values from the same system, and to understand the mechanisms that may underlie the system.

Mathematically, linear least squares is the problem of approximately solving an overdetermined system of linear equations A x = b, where b is not an element of the column space of the matrix A. The approximate solution is realized as an exact solution to A x = b', where b' is the projection of b onto the column space of A. The best approximation is then that which minimizes the sum of squared differences between the data values and their corresponding modeled values. The approach is called linear least squares since the assumed function is linear in the parameters to be estimated. Linear least squares problems are convex and have a closed-form solution that is unique, provided that the number of data points used for fitting equals or exceeds the number of unknown parameters, except in special degenerate situations. In contrast, non-linear least squares problems generally must be solved by an iterative procedure, and the problems can be non-convex with multiple optima for the objective function. If prior distributions are available, then even an underdetermined system can be solved using the Bayesian MMSE estimator.

In statistics, linear least squares problems correspond to a particularly important type of statistical model called linear regression which arises as a particular form of regression analysis. One basic form of such a model is an ordinary least squares model. The present article concentrates on the mathematical aspects of linear least squares problems, with discussion of the formulation and interpretation of statistical regression models and statistical inferences related to these being dealt with in the articles just mentioned. See outline of regression analysis for an outline of the topic.

If the experimental errors, ${\displaystyle \varepsilon }$, are uncorrelated, have a mean of zero and a constant variance, ${\displaystyle \sigma }$, the Gauss–Markov theorem states that the least-squares estimator, ${\displaystyle {\hat {\boldsymbol {\beta }}}}$, has the minimum variance of all estimators that are linear combinations of the observations. In this sense it is the best, or optimal, estimator of the parameters. Note particularly that this property is independent of the statistical distribution function of the errors. In other words, the distribution function of the errors need not be a normal distribution. However, for some probability distributions, there is no guarantee that the least-squares solution is even possible given the observations; still, in such cases it is the best estimator that is both linear and unbiased.

For example, it is easy to show that the arithmetic mean of a set of measurements of a quantity is the least-squares estimator of the value of that quantity. If the conditions of the Gauss–Markov theorem apply, the arithmetic mean is optimal, whatever the distribution of errors of the measurements might be.

However, in the case that the experimental errors do belong to a normal distribution, the least-squares estimator is also a maximum likelihood estimator.[9]

These properties underpin the use of the method of least squares for all types of data fitting, even when the assumptions are not strictly valid.

• Polynomial fitting: models are polynomials in an independent variable, x:
  • Straight line: ${\displaystyle f(x,{\boldsymbol {\beta }})=\beta _{1}+\beta _{2}x}$.[12]
  • Quadratic: ${\displaystyle f(x,{\boldsymbol {\beta }})=\beta _{1}+\beta _{2}x+\beta _{3}x^{2}}$.
  • Cubic, quartic and higher polynomials. For regression with high-order polynomials, the use of orthogonal polynomials is recommended.[13]
• Numerical smoothing and differentiation — this is an application of polynomial fitting.
• Multinomials in more than one independent variable, including surface fitting
• Curve fitting with B-splines[10]
• Chemometrics, Calibration curve, Standard addition, Gran plot, analysis of mixtures

Uses in data fitting

The primary application of linear least squares is in data fitting. Given a set of m data points ${\displaystyle y_{1},y_{2},\dots ,y_{m},}$ consisting of experimentally measured values taken at m values ${\displaystyle x_{1},x_{2},\dots ,x_{m}}$ of an independent variable (${\displaystyle x_{i}}$ may be scalar or vector quantities), and given a model function ${\displaystyle y=f(x,{\boldsymbol {\beta }}),}$ with ${\displaystyle {\boldsymbol {\beta }}=(\beta _{1},\beta _{2},\dots ,\beta _{n}),}$ it is desired to find the parameters ${\displaystyle \beta _{j}}$ such that the model function "best" fits the data. In linear least squares, linearity is meant to be with respect to parameters ${\displaystyle \beta _{j},}$ so

$${\displaystyle f(x,{\boldsymbol {\beta }})=\sum _{j=1}^{n}\beta _{j}\varphi _{j}(x).}$$

Here, the functions ${\displaystyle \varphi _{j}}$ may be nonlinear with respect to the variable x.

Ideally, the model function fits the data exactly, so

$${\displaystyle y_{i}=f(x_{i},{\boldsymbol {\beta }})}$$

for all ${\displaystyle i=1,2,\dots ,m.}$ This is usually not possible in practice, as there are more data points than there are parameters to be determined. The approach chosen then is to find the minimal possible value of the sum of squares of the residuals

$${\displaystyle r_{i}({\boldsymbol {\beta }})=y_{i}-f(x_{i},{\boldsymbol {\beta }}),\ (i=1,2,\dots ,m)}$$

so to minimize the function

$${\displaystyle S({\boldsymbol {\beta }})=\sum _{i=1}^{m}r_{i}^{2}({\boldsymbol {\beta }}).}$$

After substituting for ${\displaystyle r_{i}}$ and then for ${\displaystyle f}$, this minimization problem becomes the quadratic minimization problem above with

$${\displaystyle X_{ij}=\varphi _{j}(x_{i}),}$$

and the best fit can be found by solving the normal equations.

A plot of the data points (in red), the least squares line of best fit (in blue), and the residuals (in green)

As a result of an experiment, four ${\displaystyle (x,y)}$ data points were obtained, ${\displaystyle (1,6),}$ ${\displaystyle (2,5),}$ ${\displaystyle (3,7),}$ and ${\displaystyle (4,10)}$ (shown in red in the diagram on the right). We hope to find a line ${\displaystyle y=\beta _{1}+\beta _{2}x}$ that best fits these four points. In other words, we would like to find the numbers ${\displaystyle \beta _{1}}$ and ${\displaystyle \beta _{2}}$ that approximately solve the overdetermined linear system:

$${\displaystyle {\begin{alignedat}{3}\beta _{1}+1\beta _{2}+r_{1}&&\;=\;&&6&\\\beta _{1}+2\beta _{2}+r_{2}&&\;=\;&&5&\\\beta _{1}+3\beta _{2}+r_{3}&&\;=\;&&7&\\\beta _{1}+4\beta _{2}+r_{4}&&\;=\;&&10&\\\end{alignedat}}}$$

of four equations in two unknowns in some "best" sense.

${\displaystyle r}$ represents the residual, at each point, between the curve fit and the data:

$${\displaystyle {\begin{alignedat}{3}r_{1}&&\;=\;&&6-(\beta _{1}+1\beta _{2})&\\r_{2}&&\;=\;&&5-(\beta _{1}+2\beta _{2})&\\r_{3}&&\;=\;&&7-(\beta _{1}+3\beta _{2})&\\r_{4}&&\;=\;&&10-(\beta _{1}+4\beta _{2})&\\\end{alignedat}}}$$

The least squares approach to solving this problem is to try to make the sum of the squares of these residuals as small as possible; that is, to find the minimum of the function:

$${\displaystyle {\begin{aligned}S(\beta _{1},\beta _{2})&=r_{1}^{2}+r_{2}^{2}+r_{3}^{2}+r_{4}^{2}\\[6pt]&=[6-(\beta _{1}+1\beta _{2})]^{2}+[5-(\beta _{1}+2\beta _{2})]^{2}+[7-(\beta _{1}+3\beta _{2})]^{2}+[10-(\beta _{1}+4\beta _{2})]^{2}\\[6pt]&=4\beta _{1}^{2}+30\beta _{2}^{2}+20\beta _{1}\beta _{2}-56\beta _{1}-154\beta _{2}+210\\[6pt]\end{aligned}}}$$

The minimum is determined by calculating the partial derivatives of ${\displaystyle S(\beta _{1},\beta _{2})}$ with respect to ${\displaystyle \beta _{1}}$ and ${\displaystyle \beta _{2}}$ and setting them to zero:

$${\displaystyle {\frac {\partial S}{\partial \beta _{1}}}=0=8\beta _{1}+20\beta _{2}-56}$$

$${\displaystyle {\frac {\partial S}{\partial \beta _{2}}}=0=20\beta _{1}+60\beta _{2}-154.}$$

This results in a system of two equations in two unknowns, called the normal equations, which when solved give:

$${\displaystyle \beta _{1}=3.5}$$

$${\displaystyle \beta _{2}=1.4}$$

and the equation ${\displaystyle y=3.5+1.4x}$ is the line of best fit. The residuals, that is, the differences between the ${\displaystyle y}$ values from the observations and the ${\displaystyle y}$ predicated variables by using the line of best fit, are then found to be ${\displaystyle 1.1,}$ ${\displaystyle -1.3,}$ ${\displaystyle -0.7,}$ and ${\displaystyle 0.9}$ (see the diagram on the right). The minimum value of the sum of squares of the residuals is ${\displaystyle S(3.5,1.4)=1.1^{2}+(-1.3)^{2}+(-0.7)^{2}+0.9^{2}=4.2.}$

More generally, one can have ${\displaystyle n}$ regressors ${\displaystyle x_{j}}$, and a linear model

$${\displaystyle y=\beta _{0}+\sum _{j=1}^{n}\beta _{j}x_{j}.}$$

Using a quadratic model

The result of fitting a quadratic function ${\displaystyle y=\beta _{1}+\beta _{2}x+\beta _{3}x^{2}\,}$ (in blue) through a set of data points ${\displaystyle (x_{i},y_{i})}$ (in red). In linear least squares the function need not be linear in the argument ${\displaystyle x,}$ but only in the parameters ${\displaystyle \beta _{j}}$ that are determined to give the best fit.

Importantly, in "linear least squares", we are not restricted to using a line as the model as in the above example. For instance, we could have chosen the restricted quadratic model ${\displaystyle y=\beta _{1}x^{2}}$. This model is still linear in the ${\displaystyle \beta _{1}}$ parameter, so we can still perform the same analysis, constructing a system of equations from the data points:

$${\displaystyle {\begin{alignedat}{2}6&&\;=\beta _{1}(1)^{2}+r_{1}\\5&&\;=\beta _{1}(2)^{2}+r_{2}\\7&&\;=\beta _{1}(3)^{2}+r_{3}\\10&&\;=\beta _{1}(4)^{2}+r_{4}\\\end{alignedat}}}$$

The partial derivatives with respect to the parameters (this time there is only one) are again computed and set to 0:

$${\displaystyle {\frac {\partial S}{\partial \beta _{1}}}=0=708\beta _{1}-498}$$

and solved

$${\displaystyle \beta _{1}=0.703}$$

leading to the resulting best fit model ${\displaystyle y=0.703x^{2}.}$

• Line-line intersection#Nearest point to non-intersecting lines, an application
• Line fitting
• Nonlinear least squares
• Regularized least squares
• Simple linear regression
• Partial least squares regression
• Linear function

• Bevington, Philip R.; Robinson, Keith D. (2003). Data Reduction and Error Analysis for the Physical Sciences. McGraw-Hill. ISBN 978-0-07-247227-1.

• Least Squares Fitting – From MathWorld
• Least Squares Fitting-Polynomial – From MathWorld