Conditional independence

Let ${\displaystyle A}$, ${\displaystyle B}$, and ${\displaystyle C}$ be events. ${\displaystyle A}$ and ${\displaystyle B}$ are said to be conditionally independent given ${\displaystyle C}$ if and only if ${\displaystyle P(C)>0}$ and:

${\displaystyle P(A\mid B,C)=P(A\mid C)}$

This property is often written: ${\displaystyle (A\perp \!\!\!\perp B\mid C)}$, which should be read ${\displaystyle ((A\perp \!\!\!\perp B)\vert C)}$.

Equivalently, conditional independence may be stated as:

${\displaystyle P(A,B|C)=P(A|C)P(B|C)}$

where ${\displaystyle P(A,B|C)}$ is the joint probability of ${\displaystyle A}$ and ${\displaystyle B}$ given ${\displaystyle C}$. This alternate formulation states that ${\displaystyle A}$ and ${\displaystyle B}$ are independent events, given ${\displaystyle C}$.

It demonstrates that ${\displaystyle (A\perp \!\!\!\perp B\mid C)}$ is equivalent to ${\displaystyle (B\perp \!\!\!\perp A\mid C)}$.

Two discrete random variables ${\displaystyle X}$ and ${\displaystyle Y}$ are conditionally independent given a third discrete random variable ${\displaystyle Z}$ if and only if they are independent in their conditional probability distribution given ${\displaystyle Z}$. That is, ${\displaystyle X}$ and ${\displaystyle Y}$ are conditionally independent given ${\displaystyle Z}$ if and only if, given any value of ${\displaystyle Z}$, the probability distribution of ${\displaystyle X}$ is the same for all values of ${\displaystyle Y}$ and the probability distribution of ${\displaystyle Y}$ is the same for all values of ${\displaystyle X}$. Formally:

${\displaystyle (X\perp \!\!\!\perp Y)\mid Z\quad \iff \quad F_{X,Y\,\mid \,Z\,=\,z}(x,y)=F_{X\,\mid \,Z\,=\,z}(x)\cdot F_{Y\,\mid \,Z\,=\,z}(y)\quad {\text{for all }}x,y,z}$

(Eq.2)

where ${\displaystyle F_{X,Y\,\mid \,Z\,=\,z}(x,y)=\Pr(X\leq x,Y\leq y\mid Z=z)}$ is the conditional cumulative distribution function of ${\displaystyle X}$ and ${\displaystyle Y}$ given ${\displaystyle Z}$.

Two events ${\displaystyle R}$ and ${\displaystyle B}$ are conditionally independent given a σ-algebra ${\displaystyle \Sigma }$ if

${\displaystyle \Pr(R,B\mid \Sigma )=\Pr(R\mid \Sigma )\Pr(B\mid \Sigma ){\text{ a.s.}}}$

where ${\displaystyle \Pr(A\mid \Sigma )}$ denotes the conditional expectation of the indicator function of the event ${\displaystyle A}$, ${\displaystyle \chi _{A}}$, given the sigma algebra ${\displaystyle \Sigma }$. That is,

${\displaystyle \Pr(A\mid \Sigma ):=\operatorname {E} [\chi _{A}\mid \Sigma ].}$

Two random variables ${\displaystyle X}$ and ${\displaystyle Y}$ are conditionally independent given a σ-algebra ${\displaystyle \Sigma }$ if the above equation holds for all ${\displaystyle R}$ in ${\displaystyle \sigma (X)}$ and ${\displaystyle B}$ in ${\displaystyle \sigma (Y)}$.

Two random variables ${\displaystyle X}$ and ${\displaystyle Y}$ are conditionally independent given a random variable ${\displaystyle W}$ if they are independent given σ(W): the σ-algebra generated by ${\displaystyle W}$. This is commonly written:

${\displaystyle X\perp \!\!\!\perp Y\mid W}$ or

${\displaystyle X\perp Y\mid W}$

This it read "${\displaystyle X}$ is independent of ${\displaystyle Y}$, given ${\displaystyle W}$"; the conditioning applies to the whole statement: "(${\displaystyle X}$ is independent of ${\displaystyle Y}$) given ${\displaystyle W}$".

${\displaystyle (X\perp \!\!\!\perp Y)\mid W}$

This notation extends ${\displaystyle X\perp \!\!\!\perp Y}$ for "${\displaystyle X}$ is independent of ${\displaystyle Y}$."

If ${\displaystyle W}$ assumes a countable set of values, this is equivalent to the conditional independence of X and Y for the events of the form ${\displaystyle [W=w]}$. Conditional independence of more than two events, or of more than two random variables, is defined analogously.

The following two examples show that ${\displaystyle X\perp \!\!\!\perp Y}$ neither implies nor is implied by ${\displaystyle (X\perp \!\!\!\perp Y)\mid W}$.

First, suppose ${\displaystyle W}$ is 0 with probability 0.5 and 1 otherwise. When W = 0 take ${\displaystyle X}$ and ${\displaystyle Y}$ to be independent, each having the value 0 with probability 0.99 and the value 1 otherwise. When ${\displaystyle W=1}$, ${\displaystyle X}$ and ${\displaystyle Y}$ are again independent, but this time they take the value 1 with probability 0.99. Then ${\displaystyle (X\perp \!\!\!\perp Y)\mid W}$. But ${\displaystyle X}$ and ${\displaystyle Y}$ are dependent, because Pr(X = 0) < Pr(X = 0|Y = 0). This is because Pr(X = 0) = 0.5, but if Y = 0 then it's very likely that W = 0 and thus that X = 0 as well, so Pr(X = 0|Y = 0) > 0.5.

For the second example, suppose ${\displaystyle X\perp \!\!\!\perp Y}$, each taking the values 0 and 1 with probability 0.5. Let ${\displaystyle W}$ be the product ${\displaystyle X\cdot Y}$. Then when ${\displaystyle W=0}$, Pr(X = 0) = 2/3, but Pr(X = 0|Y = 0) = 1/2, so ${\displaystyle (X\perp \!\!\!\perp Y)\mid W}$ is false. This is also an example of Explaining Away. See Kevin Murphy's tutorial [3] where ${\displaystyle X}$ and ${\displaystyle Y}$ take the values "brainy" and "sporty".

Two random vectors ${\displaystyle \mathbf {X} =(X_{1},\ldots ,X_{l})^{\mathrm {T} }}$ and ${\displaystyle \mathbf {Y} =(Y_{1},\ldots ,Y_{m})^{\mathrm {T} }}$ are conditionally independent given a third random vector ${\displaystyle \mathbf {Z} =(Z_{1},\ldots ,Z_{n})^{\mathrm {T} }}$ if and only if they are independent in their conditional cumulative distribution given ${\displaystyle \mathbf {Z} }$. Formally:

${\displaystyle (\mathbf {X} \perp \!\!\!\perp \mathbf {Y} )\mid \mathbf {Z} \quad \iff \quad F_{\mathbf {X} ,\mathbf {Y} |\mathbf {Z} =\mathbf {z} }(\mathbf {x} ,\mathbf {y} )=F_{\mathbf {X} \,\mid \,\mathbf {Z} \,=\,\mathbf {z} }(\mathbf {x} )\cdot F_{\mathbf {Y} \,\mid \,\mathbf {Z} \,=\,\mathbf {z} }(\mathbf {y} )\quad {\text{for all }}\mathbf {x} ,\mathbf {y} ,\mathbf {z} }$

(Eq.3)

where ${\displaystyle \mathbf {x} =(x_{1},\ldots ,x_{l})^{\mathrm {T} }}$, ${\displaystyle \mathbf {y} =(y_{1},\ldots ,y_{m})^{\mathrm {T} }}$ and ${\displaystyle \mathbf {z} =(z_{1},\ldots ,z_{n})^{\mathrm {T} }}$ and the conditional cumulative distributions are defined as follows.

${\displaystyle {\begin{aligned}F_{\mathbf {X} ,\mathbf {Y} \,\mid \,\mathbf {Z} \,=\,\mathbf {z} }(\mathbf {x} ,\mathbf {y} )&=\Pr(X_{1}\leq x_{1},\ldots ,X_{l}\leq x_{l},Y_{1}\leq y_{1},\ldots ,Y_{m}\leq y_{m}\mid Z_{1}=z_{1},\ldots ,Z_{n}=z_{n})\\[6pt]F_{\mathbf {X} \,\mid \,\mathbf {Z} \,=\,\mathbf {z} }(\mathbf {x} )&=\Pr(X_{1}\leq x_{1},\ldots ,X_{l}\leq x_{l}\mid Z_{1}=z_{1},\ldots ,Z_{n}=z_{n})\\[6pt]F_{\mathbf {Y} \,\mid \,\mathbf {Z} \,=\,\mathbf {z} }(\mathbf {y} )&=\Pr(Y_{1}\leq y_{1},\ldots ,Y_{m}\leq y_{m}\mid Z_{1}=z_{1},\ldots ,Z_{n}=z_{n})\end{aligned}}}$

Let p be the proportion of voters who will vote "yes" in an upcoming referendum. In taking an opinion poll, one chooses n voters randomly from the population. For i = 1, ..., n, let X_i = 1 or 0 corresponding, respectively, to whether or not the ith chosen voter will or will not vote "yes".

In a frequentist approach to statistical inference one would not attribute any probability distribution to p (unless the probabilities could be somehow interpreted as relative frequencies of occurrence of some event or as proportions of some population) and one would say that X₁, ..., X_n are independent random variables.

By contrast, in a Bayesian approach to statistical inference, one would assign a probability distribution to p regardless of the non-existence of any such "frequency" interpretation, and one would construe the probabilities as degrees of belief that p is in any interval to which a probability is assigned. In that model, the random variables X₁, ..., X_n are not independent, but they are conditionally independent given the value of p. In particular, if a large number of the Xs are observed to be equal to 1, that would imply a high conditional probability, given that observation, that p is near 1, and thus a high conditional probability, given that observation, that the next X to be observed will be equal to 1.

A set of rules governing statements of conditional independence have been derived from the basic definition.[4][5]

These rules were termed "Graphoid Axioms" by Pearl and Paz,[6] because they hold in graphs, where ${\displaystyle X\perp \!\!\!\perp A\mid B}$ is interpreted to mean: "All paths from X to A are intercepted by the set B".[7]

• Graphoid
• Conditional dependence
• de Finetti's theorem
• Conditional expectation

• Media related to Conditional independence at Wikimedia Commons