Bayes' Rule

In probability theory and statistics, Bayes' theorem (or Bayes' rule ) is a result that is of importance in the mathematical manipulation of conditional probabilities. It is a result that derives from the more basic axioms of probability. When applied, the probabilities involved in Bayes' theorem may have any of a number of probability interpretations. In one of these interpretations, the theorem is used directly as part of a particular approach to statistical inference. In particular, with the Bayesian interpretation of probability, the theorem expresses how a subjective degree of belief should rationally change to account for evidence. This is known as Bayesian inference, which is fundamental to Bayesian statistics.

Bayes' rule relates the odds of event $A_1$ to event $A_2$, before (prior to) and after (posterior to) conditioning on another event $B$. The odds on $A_1$ to event $A_2$ is simply the ratio of the probabilities of the two events. The relationship is expressed in terms of the likelihood ratio, or Bayes' factor. By definition, this is the ratio of the conditional probabilities of the event $B$ given that $A_1$ is the case or that $A_2$ is the case, respectively. The rule simply states:

Posterior odds equals prior odds times Bayes' factor.

More specifically, given events $A_1$, $A_2$ and $B$, Bayes' rule states that the conditional odds of $A_1:A_2$ given $B$ are equal to the marginal odds $A_1:A_2$ multiplied by the Bayes factor or likelihood ratio. This is shown in the following formulas:

$O(A_1:A_2|B) = \Lambda(A_1:A_2|B)\cdot O(A_1:A_2)$

Where the likelihood ratio $\Lambda$ is the ratio of the conditional probabilities of the event $B$ given that $A_1$ is the case or that $A_2$ is the case, respectively:

$\displaystyle \Lambda(A_1:A_2|B) = \frac{P(B|A_1)}{P(B|A_2)}$

Bayes' rule is widely used in statistics, science and engineering, such as in: model selection, probabilistic expert systems based on Bayes' networks, statistical proof in legal proceedings, email spam filters, etc. Bayes' rule tells us how unconditional and conditional probabilities are related whether we work with a frequentist or a Bayesian interpretation of probability. Under the Bayesian interpretation it is frequently applied in the situation where $A_1$ and $A_2$ are competing hypotheses, and $B$ is some observed evidence. The rule shows how one's judgement on whether $A_1$ or $A_2$ is true should be updated on observing the evidence.

Bayesian Inference

Bayesian inference is a method of inference in which Bayes' rule is used to update the probability estimate for a hypothesis as additional evidence is learned. Bayesian updating is an important technique throughout statistics, and especially in mathematical statistics. Bayesian updating is especially important in the dynamic analysis of a sequence of data. Bayesian inference has found application in a range of fields including science, engineering, philosophy, medicine, and law.

Informal Definition

Rationally, Bayes' rule makes a great deal of sense. If the evidence does not match up with a hypothesis, one should reject the hypothesis. But if a hypothesis is extremely unlikely a priori, one should also reject it, even if the evidence does appear to match up.

For example, imagine that we have various hypotheses about the nature of a newborn baby of a friend, including:

• $H_1$: The baby is a brown-haired boy.
• $H_2$: The baby is a blond-haired girl.
• $H_3$: The baby is a dog.

Then, consider two scenarios:

1. We're presented with evidence in the form of a picture of a blond-haired baby girl. We find this evidence supports $H_2$ and opposes $H_1$ and $H_3$.
2. We're presented with evidence in the form of a picture of a baby dog. Although this evidence, treated in isolation, supports $H_3$, my prior belief in this hypothesis (that a human can give birth to a dog) is extremely small. Therefore, the posterior probability is nevertheless small.

The critical point about Bayesian inference, then, is that it provides a principled way of combining new evidence with prior beliefs, through the application of Bayes' rule. Furthermore, Bayes' rule can be applied iteratively. After observing some evidence, the resulting posterior probability can then be treated as a prior probability, and a new posterior probability computed from new evidence. This allows for Bayesian principles to be applied to various kinds of evidence, whether viewed all at once or over time. This procedure is termed Bayesian updating.

Bayes' Theorem

A blue neon sign at the Autonomy Corporation in Cambridge, showing the simple statement of Bayes' theorem.