Kraft–McMillan inequality

Example for binary tree. Red nodes represent a prefix tree. The method for calculating the number of descendant leaf nodes in the full tree is shown.

First, let us show that the Kraft inequality holds whenever the code for ${\displaystyle S}$ is a prefix code.

Suppose that ${\displaystyle \ell _{1}\leqslant \ell _{2}\leqslant \cdots \leqslant \ell _{n}}$. Let ${\displaystyle A}$ be the full ${\displaystyle r}$-ary tree of depth ${\displaystyle \ell _{n}}$ (thus, every node of ${\displaystyle A}$ at level ${\displaystyle <\ell _{n}}$ has ${\displaystyle r}$ children, while the nodes at level ${\displaystyle \ell _{n}}$ are leaves). Every word of length ${\displaystyle \ell \leqslant \ell _{n}}$ over an ${\displaystyle r}$-ary alphabet corresponds to a node in this tree at depth ${\displaystyle \ell }$. The ${\displaystyle i}$th word in the prefix code corresponds to a node ${\displaystyle v_{i}}$; let ${\displaystyle A_{i}}$ be the set of all leaf nodes (i.e. of nodes at depth ${\displaystyle \ell _{n}}$) in the subtree of ${\displaystyle A}$ rooted at ${\displaystyle v_{i}}$. That subtree being of height ${\displaystyle \ell _{n}-\ell _{i}}$, we have

${\displaystyle |A_{i}|=r^{\ell _{n}-\ell _{i}}.}$

Since the code is a prefix code, those subtrees cannot share any leaves, which means that

${\displaystyle A_{i}\cap A_{j}=\varnothing ,\quad i\neq j.}$

Thus, given that the total number of nodes at depth ${\displaystyle \ell _{n}}$ is ${\displaystyle r^{\ell _{n}}}$, we have

${\displaystyle \left|\bigcup _{i=1}^{n}A_{i}\right|=\sum _{i=1}^{n}|A_{i}|=\sum _{i=1}^{n}r^{\ell _{n}-\ell _{i}}\leqslant r^{\ell _{n}}}$

from which the result follows.

Conversely, given any ordered sequence of ${\displaystyle n}$ natural numbers,

${\displaystyle \ell _{1}\leqslant \ell _{2}\leqslant \cdots \leqslant \ell _{n}}$

satisfying the Kraft inequality, one can construct a prefix code with codeword lengths equal to each ${\displaystyle \ell _{i}}$ by choosing a word of length ${\displaystyle \ell _{i}}$ arbitrarily, then ruling out all words of greater length that have it as a prefix. There again, we shall interpret this in terms of leaf nodes of an ${\displaystyle r}$-ary tree of depth ${\displaystyle \ell _{n}}$. First choose any node from the full tree at depth ${\displaystyle \ell _{1}}$; it corresponds to the first word of our new code. Since we are building a prefix code, all the descendants of this node (i.e., all words that have this first word as a prefix) become unsuitable for inclusion in the code. We consider the descendants at depth ${\displaystyle \ell _{n}}$ (i.e., the leaf nodes among the descendants); there are ${\displaystyle r^{\ell _{n}-\ell _{1}}}$ such descendant nodes that are removed from consideration. The next iteration picks a (surviving) node at depth ${\displaystyle \ell _{2}}$ and removes ${\displaystyle r^{\ell _{n}-\ell _{2}}}$ further leaf nodes, and so on. After ${\displaystyle n}$ iterations, we have removed a total of

${\displaystyle \sum _{i=1}^{n}r^{\ell _{n}-\ell _{i}}}$

nodes. The question is whether we need to remove more leaf nodes than we actually have available — ${\displaystyle r^{\ell _{n}}}$ in all — in the process of building the code. Since the Kraft inequality holds, we have indeed

${\displaystyle \sum _{i=1}^{n}r^{\ell _{n}-\ell _{i}}\leqslant r^{\ell _{n}}}$

and thus a prefix code can be built. Note that as the choice of nodes at each step is largely arbitrary, many different suitable prefix codes can be built, in general.

Now we will prove that the Kraft inequality holds whenever ${\displaystyle S}$ is a uniquely decodable code. (The converse needs not be proven, since we have already proven it for prefix codes, which is a stronger claim.)

Denote ${\displaystyle C=\sum _{i=1}^{n}r^{-l_{i}}}$. The idea of the proof is to get an upper bound on ${\displaystyle C^{m}}$ for ${\displaystyle m\in \mathbb {N} }$ and show that it can only hold for all ${\displaystyle m}$ if ${\displaystyle C\leq 1}$. Rewrite ${\displaystyle C^{m}}$ as

${\displaystyle {\begin{aligned}C^{m}&=\left(\sum _{i=1}^{n}r^{-l_{i}}\right)^{m}\\&=\sum _{i_{1}=1}^{n}\sum _{i_{2}=1}^{n}\cdots \sum _{i_{m}=1}^{n}r^{-\left(l_{i_{1}}+l_{i_{2}}+\cdots +l_{i_{m}}\right)}\\\end{aligned}}}$

Consider all m-powers ${\displaystyle S^{m}}$, in the form of words ${\displaystyle s_{i_{1}}s_{i_{2}}\dots s_{i_{m}}}$, where ${\displaystyle i_{1},i_{2},\dots ,i_{m}}$ are indices between 1 and ${\displaystyle n}$. Note that, since S was assumed to uniquely decodable, ${\displaystyle s_{i_{1}}s_{i_{2}}\dots s_{i_{m}}=s_{j_{1}}s_{j_{2}}\dots s_{j_{m}}}$ implies ${\displaystyle i_{1}=j_{1},i_{2}=j_{2},\dots ,i_{m}=j_{m}}$. This means that each summand corresponds to exactly one word in ${\displaystyle S^{m}}$. This allows us to rewrite the equation to

${\displaystyle C^{m}=\sum _{\ell =1}^{m\cdot \ell _{max}}q_{\ell }\,r^{-\ell }}$

where ${\displaystyle q_{\ell }}$ is the number of codewords in ${\displaystyle S^{m}}$ of length ${\displaystyle \ell }$ and ${\displaystyle \ell _{max}}$ is the length of the longest codeword in ${\displaystyle S}$. For an ${\displaystyle r}$-letter alphabet there are only ${\displaystyle r^{\ell }}$ possible words of length ${\displaystyle \ell }$, so ${\displaystyle q_{\ell }\leq r^{\ell }}$. Using this, we upper bound ${\displaystyle C^{m}}$:

${\displaystyle {\begin{aligned}C^{m}&=\sum _{\ell =1}^{m\cdot \ell _{max}}q_{\ell }\,r^{-\ell }\\&\leq \sum _{\ell =1}^{m\cdot \ell _{max}}r^{\ell }\,r^{-\ell }=m\cdot \ell _{max}\end{aligned}}}$

Taking the ${\displaystyle m}$-th root, we get

${\displaystyle C=\sum _{i=1}^{n}r^{-l_{i}}\leq \left(m\cdot \ell _{max}\right)^{\frac {1}{m}}}$

This bound holds for any ${\displaystyle m\in \mathbb {N} }$. The right side is 1 asymptotically, so ${\displaystyle \sum _{i=1}^{n}r^{-l_{i}}\leq 1}$ must hold (otherwise the inequality would be broken for a large enough ${\displaystyle m}$).

Given a sequence of ${\displaystyle n}$ natural numbers,

${\displaystyle \ell _{1}\leqslant \ell _{2}\leqslant \cdots \leqslant \ell _{n}}$

satisfying the Kraft inequality, we can construct a prefix code as follows. Define the i^th codeword, C_i, to be the first ${\displaystyle \ell _{i}}$ digits after the radix point (e.g. decimal point) in the base r representation of

${\displaystyle \sum _{j=1}^{i-1}r^{-\ell _{j}}.}$

Note that by Kraft's inequality, this sum is never more than 1. Hence the codewords capture the entire value of the sum. Therefore, for j > i, the first ${\displaystyle \ell _{i}}$ digits of C_j form a larger number than C_i, so the code is prefix free.