Independent and identically distributed random variables

Suppose that the random variables ${\displaystyle X}$ and ${\displaystyle Y}$ are defined to assume values in ${\displaystyle I\subseteq \mathbb {R} }$. Let ${\displaystyle F_{X}(x)=\operatorname {P} (X\leq x)}$ and ${\displaystyle F_{Y}(y)=\operatorname {P} (Y\leq y)}$ be the cumulative distribution functions of ${\displaystyle X}$ and ${\displaystyle Y}$, respectively, and denote their joint cumulative distribution function by ${\displaystyle F_{X,Y}(x,y)=\operatorname {P} (X\leq x\land Y\leq y)}$.

Two random variables ${\displaystyle X}$ and ${\displaystyle Y}$ are identically distributed if and only if[6] ${\displaystyle F_{X}(x)=F_{Y}(x)\,\forall x\in I}$.

Two random variables ${\displaystyle X}$ and ${\displaystyle Y}$ are independent if and only if ${\displaystyle F_{X,Y}(x,y)=F_{X}(x)\cdot F_{Y}(y)\,\forall x,y\in I}$. (See further Independence (probability theory) § Two random variables.)

Two random variables ${\displaystyle X}$ and ${\displaystyle Y}$ are i.i.d. if they are independent and identically distributed, i.e. if and only if

${\displaystyle {\begin{aligned}&F_{X}(x)=F_{Y}(x)\,&\forall x\in I\\&F_{X,Y}(x,y)=F_{X}(x)\cdot F_{Y}(y)\,&\forall x,y\in I\end{aligned}}}$

(Eq.1)

The definition extends naturally to more than two random variables. We say that ${\displaystyle n}$ random variables ${\displaystyle X_{1},\ldots ,X_{n}}$ are i.i.d. if they are independent (see further Independence (probability theory) § More than two random variables) and identically distributed, i.e. if and only if

${\displaystyle {\begin{aligned}&F_{X_{1}}(x)=F_{X_{k}}(x)\,&\forall k\in \{1,\ldots ,n\}{\text{ and }}\forall x\in I\\&F_{X_{1},\ldots ,X_{n}}(x_{1},\ldots ,x_{n})=F_{X_{1}}(x_{1})\cdot \ldots \cdot F_{X_{n}}(x_{n})\,&\forall x_{1},\ldots ,x_{n}\in I\end{aligned}}}$

(Eq.2)

where ${\displaystyle F_{X_{1},\ldots ,X_{n}}(x_{1},\ldots ,x_{n})=\operatorname {P} (X_{1}\leq x_{1}\land \ldots \land X_{n}\leq x_{n})}$ denotes the joint cumulative distribution function of ${\displaystyle X_{1},\ldots ,X_{n}}$.

In probability theory, two events, ${\textstyle \color {red}A}$ and ${\textstyle \definecolor {Green}{rgb}{0,0.5019607843137255,0}\definecolor {green}{rgb}{0,0.5019607843137255,0}\color {Green}B}$, are called independent if and only if ${\textstyle \definecolor {Green}{rgb}{0,0.5019607843137255,0}\definecolor {green}{rgb}{0,0.5019607843137255,0}P({\color {red}A}\ \mathrm {and} \ {\color {green}B})=P({\color {red}A})P({\color {green}B})}$. In the following, ${\textstyle \definecolor {Green}{rgb}{0,0.5019607843137255,0}\definecolor {green}{rgb}{0,0.5019607843137255,0}P({\color {red}A}{\color {green}B})}$ is short for ${\textstyle \definecolor {Green}{rgb}{0,0.5019607843137255,0}\definecolor {green}{rgb}{0,0.5019607843137255,0}P({\color {red}A}\ \mathrm {and} \ {\color {green}B})}$.

Suppose there are two events of the experiment, ${\textstyle \color {red}A}$ and ${\textstyle \definecolor {Green}{rgb}{0,0.5019607843137255,0}\definecolor {green}{rgb}{0,0.5019607843137255,0}\color {Green}B}$. If ${\textstyle P({\color {red}A})>0}$, there is a possibility ${\textstyle P({\color {green}B}|{\color {red}A})}$. Generally, the occurrence of ${\textstyle \color {red}A}$ has an effect on the probability of ${\textstyle \definecolor {Green}{rgb}{0,0.5019607843137255,0}\definecolor {green}{rgb}{0,0.5019607843137255,0}\color {Green}B}$, which is called conditional probability, and only when the occurrence of ${\textstyle \color {red}A}$ has no effect on the occurrence of ${\textstyle \definecolor {Green}{rgb}{0,0.5019607843137255,0}\definecolor {green}{rgb}{0,0.5019607843137255,0}\color {Green}B}$, there is ${\textstyle \definecolor {Green}{rgb}{0,0.5019607843137255,0}\definecolor {green}{rgb}{0,0.5019607843137255,0}P({\color {green}B}|{\color {red}A})=P({\color {green}B})}$.

Note: If ${\textstyle P({\color {red}A})>0}$ and ${\textstyle \definecolor {Green}{rgb}{0,0.5019607843137255,0}\definecolor {green}{rgb}{0,0.5019607843137255,0}P({\color {Green}B})>0}$, then ${\textstyle \color {red}A}$ and ${\textstyle \definecolor {Green}{rgb}{0,0.5019607843137255,0}\definecolor {green}{rgb}{0,0.5019607843137255,0}\color {Green}B}$ are mutually independent which cannot be established with mutually incompatible at the same time; that is, independence must be compatible and mutual exclusion must be related.

Suppose ${\textstyle \color {red}A}$, ${\textstyle \definecolor {Green}{rgb}{0,0.5019607843137255,0}\definecolor {green}{rgb}{0,0.5019607843137255,0}\color {Green}B}$, and ${\textstyle \definecolor {blue}{rgb}{0,0,1}\color {blue}C}$ are three events. If ${\textstyle \definecolor {Green}{rgb}{0,0.5019607843137255,0}\definecolor {green}{rgb}{0,0.5019607843137255,0}P({\color {red}A}{\color {green}B})=P({\color {red}A})P({\color {green}B})}$, ${\textstyle \definecolor {Green}{rgb}{0,0.5019607843137255,0}\definecolor {green}{rgb}{0,0.5019607843137255,0}\definecolor {blue}{rgb}{0,0,1}\definecolor {Blue}{rgb}{0,0,1}P({\color {green}B}{\color {blue}C})=P({\color {green}B})P({\color {blue}C})}$, ${\textstyle \definecolor {blue}{rgb}{0,0,1}P({\color {red}A}{\color {blue}C})=P({\color {red}A})P({\color {blue}C})}$, and ${\textstyle \definecolor {Green}{rgb}{0,0.5019607843137255,0}\definecolor {green}{rgb}{0,0.5019607843137255,0}\definecolor {blue}{rgb}{0,0,1}\definecolor {Blue}{rgb}{0,0,1}P({\color {red}A}{\color {green}B}{\color {blue}C})=P({\color {red}A})P({\color {green}B})P({\color {blue}C})}$ are satisfied, then the events ${\textstyle \color {red}A}$, ${\textstyle \definecolor {Green}{rgb}{0,0.5019607843137255,0}\definecolor {green}{rgb}{0,0.5019607843137255,0}\color {Green}B}$, and ${\textstyle \definecolor {blue}{rgb}{0,0,1}\color {blue}C}$ are mutually independent.

A more general definition is there are ${\textstyle n}$ events, ${\textstyle {\color {red}A}_{1},{\color {red}A}_{2},\ldots ,{\color {red}A}_{n}}$. If the probabilities of the product events for any ${\textstyle 2,3,\ldots ,n}$ events are equal to the product of the probabilities of each event, then the events ${\textstyle {\color {red}A}_{1},{\color {red}A}_{2},\ldots ,{\color {red}A}_{n}}$ are independent of each other.

A sequence of outcomes of spins of a fair or unfair roulette wheel is i.i.d. One implication of this is that if the roulette ball lands on "red", for example, 20 times in a row, the next spin is no more or less likely to be "black" than on any other spin (see the gambler's fallacy).

Toss a coin 10 times and record how many times the coin lands on heads.

1. Independent – Each outcome of landing will not affect the other outcome, which means the 10 results are independent from each other.
2. Identically distributed – Regardless of whether the coin is fair (probability 1/2 of heads) or unfair, as long as the same coin is used for each flip, each flip will have the same probability as each other flip.

Such a sequence of two possible i.i.d. outcomes is also called a Bernoulli process.

Roll a die 10 times and record how many times the result is 1.

1. Independent – Each outcome of the die roll will not affect the next one, which means the 10 results are independent from each other.
2. Identically distributed – Regardless of whether the die is fair or weighted, each roll will have the same probability as each other roll. In contrast, rolling 10 different dice, some of which are weighted and some of which are not, would not produce i.i.d. variables.

Choose a card from a standard deck of cards containing 52 cards, then place the card back in the deck. Repeat it 52 times. Record the number of kings that appear.

1. Independent – Each outcome of the card will not affect the next one, which means the 52 results are independent from each other. In contrast, if each card that is drawn is kept out of the deck, subsequent draws would be affected by it (drawing one king would make drawing a second king less likely), and the result would not be independent.
2. Identically distributed – After drawing one card from it, each time the probability for a king is 4/52, which means the probability is identical each time.

The most general notion which shares the main properties of i.i.d. variables are exchangeable random variables, introduced by Bruno de Finetti. Exchangeability means that while variables may not be independent, future ones behave like past ones – formally, any value of a finite sequence is as likely as any permutation of those values – the joint probability distribution is invariant under the symmetric group.

This provides a useful generalization – for example, sampling without replacement is not independent, but is exchangeable.

In stochastic calculus, i.i.d. variables are thought of as a discrete time Lévy process: each variable gives how much one changes from one time to another. For example, a sequence of Bernoulli trials is interpreted as the Bernoulli process. One may generalize this to include continuous time Lévy processes, and many Lévy processes can be seen as limits of i.i.d. variables—for instance, the Wiener process is the limit of the Bernoulli process.