M-estimator

For positive integer r, let ${\displaystyle ({\mathcal {X}},\Sigma )}$ and ${\displaystyle (\Theta \subset \mathbb {R} ^{r},S)}$ be measure spaces. ${\displaystyle \theta \in \Theta }$ is a vector of parameters. An M-estimator of ρ-type ${\displaystyle T}$ is defined through a measurable function ${\displaystyle \rho$ :{\mathcal {X}}\times \Theta \rightarrow \mathbb {R} } . It maps a probability distribution ${\displaystyle F}$ on ${\displaystyle {\mathcal {X}}}$ to the value ${\displaystyle T(F)\in \Theta }$ (if it exists) that minimizes ${\displaystyle \int _{\mathcal {X}}\rho (x,\theta )dF(x)}$:

${\displaystyle T(F):=\arg \min _{\theta \in \Theta }\int _{\mathcal {X}}\rho (x,\theta )dF(x)}$

For example, for the maximum likelihood estimator, ${\displaystyle \rho (x,\theta )=-\log(f(x,\theta ))}$, where ${\displaystyle f(x,\theta )={\frac {\partial F(x,\theta )}{\partial x}}}$.

If ${\displaystyle \rho }$ is differentiable with respect to ${\displaystyle \theta }$, the computation of ${\displaystyle {\widehat {\theta }}}$ is usually much easier. An M-estimator of ψ-type T is defined through a measurable function ${\displaystyle \psi$ :{\mathcal {X}}\times \Theta \rightarrow \mathbb {R} ^{r}} . It maps a probability distribution F on ${\displaystyle {\mathcal {X}}}$ to the value ${\displaystyle T(F)\in \Theta }$ (if it exists) that solves the vector equation:

${\displaystyle \int _{\mathcal {X}}\psi (x,\theta )\,dF(x)=0}$

${\displaystyle \int _{\mathcal {X}}\psi (x,T(F))\,dF(x)=0}$

For example, for the maximum likelihood estimator, ${\displaystyle \psi (x,\theta )=\left({\frac {\partial \log(f(x,\theta ))}{\partial \theta ^{1}}},\dots ,{\frac {\partial \log(f(x,\theta ))}{\partial \theta ^{p}}}\right)^{\mathrm {T} }}$, where ${\displaystyle u^{\mathrm {T} }}$ denotes the transpose of vector u and ${\displaystyle f(x,\theta )={\frac {\partial F(x,\theta )}{\partial x}}}$.

Such an estimator is not necessarily an M-estimator of ρ-type, but if ρ has a continuous first derivative with respect to ${\displaystyle \theta }$, then a necessary condition for an M-estimator of ψ-type to be an M-estimator of ρ-type is ${\displaystyle \psi (x,\theta )=\nabla _{\theta }\rho (x,\theta )}$. The previous definitions can easily be extended to finite samples.

If the function ψ decreases to zero as ${\displaystyle x\rightarrow \pm \infty }$, the estimator is called redescending. Such estimators have some additional desirable properties, such as complete rejection of gross outliers.

In computation of M-estimators, it is sometimes useful to rewrite the objective function so that the dimension of parameters is reduced. The procedure is called “concentrating” or “profiling”. Examples in which concentrating parameters increases computation speed include seemingly unrelated regressions (SUR) models.[9] Consider the following M-estimation problem:

${\displaystyle ({\hat {\beta }}_{n},{\hat {\gamma }}_{n}):=\arg \max _{\beta ,\gamma }\textstyle \sum _{i=1}^{N}\displaystyle q(w_{i},\beta ,\gamma )}$

Assuming differentiability of the function q, M-estimator solves the first order conditions:

${\displaystyle \sum _{i=1}^{N}\triangledown _{\beta }\,q(w_{i},\beta ,\gamma )=0}$

${\displaystyle \sum _{i=1}^{N}\triangledown _{\gamma }\,q(w_{i},\beta ,\gamma )=0}$

Now, if we can solve the second equation for γ in terms of ${\displaystyle W:=(w_{1},w_{2},..,w_{N})}$ and ${\displaystyle \beta }$, the second equation becomes:

${\displaystyle \sum _{i=1}^{N}\triangledown _{\gamma }\,q(w_{i},\beta ,g(W,\beta ))=0}$

where g is, there is some function to be found. Now, we can rewrite the original objective function solely in terms of β by inserting the function g into the place of ${\displaystyle \gamma }$. As a result, there is a reduction in the number of parameters.

Whether this procedure can be done depends on particular problems at hand. However, when it is possible, concentrating parameters can facilitate computation to a great degree. For example, in estimating SUR model of 6 equations with 5 explanatory variables in each equation by Maximum Likelihood, the number of parameters declines from 51 to 30.[9]

Despite its appealing feature in computation, concentrating parameters is of limited use in deriving asymptotic properties of M-estimator.[10] The presence of W in each summand of the objective function makes it difficult to apply the law of large numbers and the central limit theorem.

It can be shown that M-estimators are asymptotically normally distributed. As such, Wald-type approaches to constructing confidence intervals and hypothesis tests can be used. However, since the theory is asymptotic, it will frequently be sensible to check the distribution, perhaps by examining the permutation or bootstrap distribution.

The influence function of an M-estimator of ${\displaystyle \psi }$-type is proportional to its defining ${\displaystyle \psi }$ function.

Let T be an M-estimator of ψ-type, and G be a probability distribution for which ${\displaystyle T(G)}$ is defined. Its influence function IF is

${\displaystyle \operatorname {IF} (x;T,G)=-{\frac {\psi (x,T(G))}{\int \left[{\frac {\partial \psi (y,\theta )}{\partial \theta }}\right]f(y)\mathrm {d} y}}}$

assuming the density function ${\displaystyle f(y)}$ exists. A proof of this property of M-estimators can be found in Huber (1981, Section 3.2).

Let (X₁, ..., X_n) be a set of independent, identically distributed random variables, with distribution F.

If we define

${\displaystyle \rho (x,\theta )={\frac {(x-\theta )^{2}}{2}},\,\!}$

we note that this is minimized when θ is the mean of the Xs. Thus the mean is an M-estimator of ρ-type, with this ρ function.

As this ρ function is continuously differentiable in θ, the mean is thus also an M-estimator of ψ-type for ψ(x, θ) = θ − x.

For the median estimation of (X₁, ..., X_n), instead we can define the ρ function as

${\displaystyle \rho (x,\theta )=|x-\theta |}$

and similarly, the ρ function is minimized when θ is the median of the Xs.

While this ρ function is not differentiable in θ, the ψ-type M-estimator, which is the subgradient of ρ function, can be expressed as

${\displaystyle \psi (x,\theta )=\operatorname {sgn}(x-\theta )}$

and

${\displaystyle \psi (x,\theta )={\begin{cases}\{-1\},&{\mbox{if }}x-\theta <0\\\{1\},&{\mbox{if }}x-\theta >0\\\left[-1,1\right],&{\mbox{if }}x-\theta =0\end{cases}}}$