Linear separability

Let ${\displaystyle X_{0}}$ and ${\displaystyle X_{1}}$ be two sets of points in an n-dimensional Euclidean space. Then ${\displaystyle X_{0}}$ and ${\displaystyle X_{1}}$ are linearly separable if there exist n + 1 real numbers ${\displaystyle w_{1},w_{2},..,w_{n},k}$, such that every point ${\displaystyle x\in X_{0}}$ satisfies ${\displaystyle \sum _{i=1}^{n}w_{i}x_{i}>k}$ and every point ${\displaystyle x\in X_{1}}$ satisfies ${\displaystyle \sum _{i=1}^{n}w_{i}x_{i}<k}$, where ${\displaystyle x_{i}}$ is the ${\displaystyle i}$-th component of ${\displaystyle x}$.

Equivalently, two sets are linearly separable precisely when their respective convex hulls are disjoint (colloquially, do not overlap).

In simple 2D, it can also be imagined that the set of points under a linear transformation collapses into a line, on which there exists a value, k, greater than which one set of points will fall into, and lesser than which the other set of points fall.

Three non-collinear points in two classes ('+' and '-') are always linearly separable in two dimensions. This is illustrated by the three examples in the following figure (the all '+' case is not shown, but is similar to the all '-' case):

However, not all sets of four points, no three collinear, are linearly separable in two dimensions. The following example would need two straight lines and thus is not linearly separable:

Notice that three points which are collinear and of the form "+ ⋅⋅⋅ — ⋅⋅⋅ +" are also not linearly separable.

A Boolean function in n variables can be thought of as an assignment of 0 or 1 to each vertex of a Boolean hypercube in n dimensions. This gives a natural division of the vertices into two sets. The Boolean function is said to be linearly separable provided these two sets of points are linearly separable. The number of distinct Boolean functions is ${\displaystyle 2^{2^{n}}}$where n is the number of variables passed into the function.[1]

H₁ does not separate the sets. H₂ does, but only with a small margin. H₃ separates them with the maximum margin.

Classifying data is a common task in machine learning. Suppose some data points, each belonging to one of two sets, are given and we wish to create a model that will decide which set a new data point will be in. In the case of support vector machines, a data point is viewed as a p-dimensional vector (a list of p numbers), and we want to know whether we can separate such points with a (p − 1)-dimensional hyperplane. This is called a linear classifier. There are many hyperplanes that might classify (separate) the data. One reasonable choice as the best hyperplane is the one that represents the largest separation, or margin, between the two sets. So we choose the hyperplane so that the distance from it to the nearest data point on each side is maximized. If such a hyperplane exists, it is known as the maximum-margin hyperplane and the linear classifier it defines is known as a maximum margin classifier.

More formally, given some training data ${\displaystyle {\mathcal {D}}}$, a set of n points of the form

${\displaystyle {\mathcal {D}}=\left\{(\mathbf {x} _{i},y_{i})\mid \mathbf {x} _{i}\in \mathbb {R} ^{p},\,y_{i}\in \{-1,1\}\right\}_{i=1}^{n}}$

where the y_i is either 1 or −1, indicating the set to which the point ${\displaystyle \mathbf {x} _{i}}$ belongs. Each ${\displaystyle \mathbf {x} _{i}}$ is a p-dimensional real vector. We want to find the maximum-margin hyperplane that divides the points having ${\displaystyle y_{i}=1}$ from those having ${\displaystyle y_{i}=-1}$. Any hyperplane can be written as the set of points ${\displaystyle \mathbf {x} }$ satisfying

${\displaystyle \mathbf {w} \cdot \mathbf {x} -b=0,}$

where ${\displaystyle \cdot }$ denotes the dot product and ${\displaystyle {\mathbf {w} }}$ the (not necessarily normalized) normal vector to the hyperplane. The parameter ${\displaystyle {\tfrac {b}{\|\mathbf {w} \|}}}$ determines the offset of the hyperplane from the origin along the normal vector ${\displaystyle {\mathbf {w} }}$.

If the training data are linearly separable, we can select two hyperplanes in such a way that they separate the data and there are no points between them, and then try to maximize their distance.

• Hyperplane separation theorem
• Kirchberger's theorem
• Perceptron
• Vapnik–Chervonenkis dimension