Outlier

Outliers can have many anomalous causes. A physical apparatus for taking measurements may have suffered a transient malfunction. There may have been an error in data transmission or transcription. Outliers arise due to changes in system behaviour, fraudulent behaviour, human error, instrument error or simply through natural deviations in populations. A sample may have been contaminated with elements from outside the population being examined. Alternatively, an outlier could be the result of a flaw in the assumed theory, calling for further investigation by the researcher. Additionally, the pathological appearance of outliers of a certain form appears in a variety of datasets, indicating that the causative mechanism for the data might differ at the extreme end (King effect).

It is proposed to determine in a series of ${\displaystyle m}$ observations the limit of error, beyond which all observations involving so great an error may be rejected, provided there are as many as ${\displaystyle n}$ such observations. The principle upon which it is proposed to solve this problem is, that the proposed observations should be rejected when the probability of the system of errors obtained by retaining them is less than that of the system of errors obtained by their rejection multiplied by the probability of making so many, and no more, abnormal observations. (Quoted in the editorial note on page 516 to Peirce (1982 edition) from A Manual of Astronomy 2:558 by Chauvenet.) [15][16][17][18]

Other methods flag observations based on measures such as the interquartile range. For example, if ${\displaystyle Q_{1}}$ and ${\displaystyle Q_{3}}$ are the lower and upper quartiles respectively, then one could define an outlier to be any observation outside the range:

${\displaystyle {\big [}Q_{1}-k(Q_{3}-Q_{1}),Q_{3}+k(Q_{3}-Q_{1}){\big ]}}$

for some nonnegative constant ${\displaystyle k}$. John Tukey proposed this test, where ${\displaystyle k=1.5}$ indicates an "outlier", and ${\displaystyle k=3}$ indicates data that is "far out".[19]

In various domains such as, but not limited to, statistics, signal processing, finance, econometrics, manufacturing, networking and data mining, the task of anomaly detection may take other approaches. Some of these may be distance-based[20][21] and density-based such as Local Outlier Factor (LOF).[22] Some approaches may use the distance to the k-nearest neighbors to label observations as outliers or non-outliers.[23]

The modified Thompson Tau test is a method used to determine if an outlier exists in a data set. The strength of this method lies in the fact that it takes into account a data set's standard deviation, average and provides a statistically determined rejection zone; thus providing an objective method to determine if a data point is an outlier.[24] How it works: First, a data set's average is determined. Next the absolute deviation between each data point and the average are determined. Thirdly, a rejection region is determined using the formula:

${\displaystyle {\text{Rejection Region}}{=}{\frac {{t_{\alpha /2}}{\left(n-1\right)}}{{\sqrt {n}}{\sqrt {n-2+{t_{\alpha /2}^{2}}}}}}}$;

where ${\displaystyle \scriptstyle {t_{\alpha /2}}}$ is the critical value from the Student t distribution with n-2 degrees of freedom, n is the sample size, and s is the sample standard deviation. To determine if a value is an outlier: Calculate ${\displaystyle \scriptstyle \delta =|(X-mean(X))/s|}$. If δ > Rejection Region, the data point is an outlier. If δ ≤ Rejection Region, the data point is not an outlier.

The modified Thompson Tau test is used to find one outlier at a time (largest value of δ is removed if it is an outlier). Meaning, if a data point is found to be an outlier, it is removed from the data set and the test is applied again with a new average and rejection region. This process is continued until no outliers remain in a data set.

Some work has also examined outliers for nominal (or categorical) data. In the context of a set of examples (or instances) in a data set, instance hardness measures the probability that an instance will be misclassified ( ${\displaystyle 1-p(y|x)}$ where y is the assigned class label and x represent the input attribute value for an instance in the training set t).[25] Ideally, instance hardness would be calculated by summing over the set of all possible hypotheses H:

${\displaystyle {\begin{aligned}IH(\langle x,y\rangle )&=\sum _{H}(1-p(y,x,h))p(h|t)\\&=\sum _{H}p(h|t)-p(y,x,h)p(h|t)\\&=1-\sum _{H}p(y,x,h)p(h|t).\end{aligned}}}$

Practically, this formulation is unfeasible as H is potentially infinite and calculating ${\displaystyle p(h|t)}$ is unknown for many algorithms. Thus, instance hardness can be approximated using a diverse subset ${\displaystyle L\subset H}$:

${\displaystyle IH_{L}(\langle x,y\rangle )=1-{\frac {1}{|L|}}\sum _{j=1}^{|L|}p(y|x,g_{j}(t,\alpha ))}$

where ${\displaystyle g_{j}(t,\alpha )}$ is the hypothesis induced by learning algorithm ${\displaystyle g_{j}}$ trained on training set t with hyperparameters ${\displaystyle \alpha }$. Instance hardness provides a continuous value for determining if an instance is an outlier instance.

Even when a normal distribution model is appropriate to the data being analyzed, outliers are expected for large sample sizes and should not automatically be discarded if that is the case.[26] Instead, one should use a method that is robust to outliers to model or analyze data with naturally occurring outliers.[26]

When deciding whether to remove an outlier, the cause has to be considered. As mentioned earlier, if the outlier's origin can be attributed to an experimental error, or if it can be otherwise determined that the outlying data point is erroneous, it is generally recommended to remove it.[26][27] However, it is more desirable to correct the erroneous value, if possible.

Removing a data point solely because it is an outlier, on the other hand, is a controversial practice, often frowned upon by many scientists and science instructors, as it typically invalidates statistical results.[26][27] While mathematical criteria provide an objective and quantitative method for data rejection, they do not make the practice more scientifically or methodologically sound, especially in small sets or where a normal distribution cannot be assumed. Rejection of outliers is more acceptable in areas of practice where the underlying model of the process being measured and the usual distribution of measurement error are confidently known.

The two common approaches to exclude outliers are truncation (or trimming) and Winsorising. Trimming discards the outliers whereas Winsorising replaces the outliers with the nearest "nonsuspect" data.[28] Exclusion can also be a consequence of the measurement process, such as when an experiment is not entirely capable of measuring such extreme values, resulting in censored data.[29]

In regression problems, an alternative approach may be to only exclude points which exhibit a large degree of influence on the estimated coefficients, using a measure such as Cook's distance.[30]

If a data point (or points) is excluded from the data analysis, this should be clearly stated on any subsequent report.

The possibility should be considered that the underlying distribution of the data is not approximately normal, having "fat tails". For instance, when sampling from a Cauchy distribution,[31] the sample variance increases with the sample size, the sample mean fails to converge as the sample size increases, and outliers are expected at far larger rates than for a normal distribution. Even a slight difference in the fatness of the tails can make a large difference in the expected number of extreme values.

A set membership approach considers that the uncertainty corresponding to the ith measurement of an unknown random vector x is represented by a set X_i (instead of a probability density function). If no outliers occur, x should belong to the intersection of all X_i's. When outliers occur, this intersection could be empty, and we should relax a small number of the sets X_i (as small as possible) in order to avoid any inconsistency.[32] This can be done using the notion of q-relaxed intersection. As illustrated by the figure, the q-relaxed intersection corresponds to the set of all x which belong to all sets except q of them. Sets X_i that do not intersect the q-relaxed intersection could be suspected to be outliers.

Figure 5. q-relaxed intersection of 6 sets for q=2 (red), q=3 (green), q= 4 (blue), q= 5 (yellow).

In cases where the cause of the outliers is known, it may be possible to incorporate this effect into the model structure, for example by using a hierarchical Bayes model, or a mixture model.[33][34]