Differential entropy

Let ${\displaystyle X}$ be a random variable with a probability density function ${\displaystyle f}$ whose support is a set ${\displaystyle {\mathcal {X}}}$. The differential entropy ${\displaystyle h(X)}$ or ${\displaystyle h(f)}$ is defined as[2]^: 243

For probability distributions which do not have an explicit density function expression, but have an explicit quantile function expression, ${\displaystyle Q(p)}$, then ${\displaystyle h(Q)}$ can be defined in terms of the derivative of ${\displaystyle Q(p)}$ i.e. the quantile density function ${\displaystyle Q'(p)}$ as[3]^: 54–59

${\displaystyle h(Q)=\int _{0}^{1}\log Q'(p)\,dp}$.

As with its discrete analog, the units of differential entropy depend on the base of the logarithm, which is usually 2 (i.e., the units are bits). See logarithmic units for logarithms taken in different bases. Related concepts such as joint, conditional differential entropy, and relative entropy are defined in a similar fashion. Unlike the discrete analog, the differential entropy has an offset that depends on the units used to measure ${\displaystyle X}$.[4]^{: 183–184} For example, the differential entropy of a quantity measured in millimeters will be log(1000) more than the same quantity measured in meters; a dimensionless quantity will have differential entropy of log(1000) more than the same quantity divided by 1000.

One must take care in trying to apply properties of discrete entropy to differential entropy, since probability density functions can be greater than 1. For example, the uniform distribution ${\displaystyle {\mathcal {U}}(0,1/2)}$ has negative differential entropy; i.e., it is better ordered than ${\displaystyle {\mathcal {U}}(0,1)}$ as shown now

${\displaystyle \int _{0}^{\frac {1}{2}}-2\log(2)\,dx=-\log(2)\,}$

being less than that of ${\displaystyle {\mathcal {U}}(0,1)}$ which has zero differential entropy. Thus, differential entropy does not share all properties of discrete entropy.

The continuous mutual information ${\displaystyle I(X;Y)}$ has the distinction of retaining its fundamental significance as a measure of discrete information since it is actually the limit of the discrete mutual information of partitions of ${\displaystyle X}$ and ${\displaystyle Y}$ as these partitions become finer and finer. Thus it is invariant under non-linear homeomorphisms (continuous and uniquely invertible maps),[5] including linear[6] transformations of ${\displaystyle X}$ and ${\displaystyle Y}$, and still represents the amount of discrete information that can be transmitted over a channel that admits a continuous space of values.

For the direct analogue of discrete entropy extended to the continuous space, see limiting density of discrete points.

• For probability densities ${\displaystyle f}$ and ${\displaystyle g}$, the Kullback–Leibler divergence ${\displaystyle D_{KL}(f||g)}$ is greater than or equal to 0 with equality only if ${\displaystyle f=g}$ almost everywhere. Similarly, for two random variables ${\displaystyle X}$ and ${\displaystyle Y}$, ${\displaystyle I(X;Y)\geq 0}$ and ${\displaystyle h(X|Y)\leq h(X)}$ with equality if and only if ${\displaystyle X}$ and ${\displaystyle Y}$ are independent.
• The chain rule for differential entropy holds as in the discrete case[2]^: 253

${\displaystyle h(X_{1},\ldots ,X_{n})=\sum _{i=1}^{n}h(X_{i}|X_{1},\ldots ,X_{i-1})\leq \sum _{i=1}^{n}h(X_{i})}$.

• Differential entropy is translation invariant, i.e. for a constant ${\displaystyle c}$.[2]^: 253

${\displaystyle h(X+c)=h(X)}$

• Differential entropy is in general not invariant under arbitrary invertible maps.

In particular, for a constant ${\displaystyle a}$

${\displaystyle h(aX)=h(X)+\log |a|}$

For a vector valued random variable ${\displaystyle \mathbf {X} }$ and an invertible (square) matrix ${\displaystyle \mathbf {A} }$

${\displaystyle h(\mathbf {A} \mathbf {X} )=h(\mathbf {X} )+\log \left(|\det \mathbf {A} |\right)}$[2]^: 253

• In general, for a transformation from a random vector to another random vector with same dimension ${\displaystyle \mathbf {Y} =m\left(\mathbf {X} \right)}$, the corresponding entropies are related via

${\displaystyle h(\mathbf {Y} )\leq h(\mathbf {X} )+\int f(x)\log \left\vert {\frac {\partial m}{\partial x}}\right\vert dx}$

where ${\displaystyle \left\vert {\frac {\partial m}{\partial x}}\right\vert }$ is the Jacobian of the transformation ${\displaystyle m}$.[7] The above inequality becomes an equality if the transform is a bijection. Furthermore, when ${\displaystyle m}$ is a rigid rotation, translation, or combination thereof, the Jacobian determinant is always 1, and ${\displaystyle h(Y)=h(X)}$.

• If a random vector ${\displaystyle X\in \mathbb {R} ^{n}}$ has mean zero and covariance matrix ${\displaystyle K}$, ${\displaystyle h(\mathbf {X} )\leq {\frac {1}{2}}\log(\det {2\pi eK})={\frac {1}{2}}\log[(2\pi e)^{n}\det {K}]}$ with equality if and only if ${\displaystyle X}$ is jointly gaussian (see below).[2]^: 254

However, differential entropy does not have other desirable properties:

• It is not invariant under change of variables, and is therefore most useful with dimensionless variables.
• It can be negative.

A modification of differential entropy that addresses these drawbacks is the relative information entropy, also known as the Kullback–Leibler divergence, which includes an invariant measure factor (see limiting density of discrete points).

Let ${\displaystyle X}$ be an exponentially distributed random variable with parameter ${\displaystyle \lambda }$, that is, with probability density function

${\displaystyle f(x)=\lambda e^{-\lambda x}{\mbox{ for }}x\geq 0.}$

Its differential entropy is then

${\displaystyle h_{e}(X)\,}$	${\displaystyle =-\int _{0}^{\infty }\lambda e^{-\lambda x}\log(\lambda e^{-\lambda x})\,dx}$
	${\displaystyle =-\left(\int _{0}^{\infty }(\log \lambda )\lambda e^{-\lambda x}\,dx+\int _{0}^{\infty }(-\lambda x)\lambda e^{-\lambda x}\,dx\right)}$
	${\displaystyle =-\log \lambda \int _{0}^{\infty }f(x)\,dx+\lambda E[X]}$
	${\displaystyle =-\log \lambda +1\,.}$

Here, ${\displaystyle h_{e}(X)}$ was used rather than ${\displaystyle h(X)}$ to make it explicit that the logarithm was taken to base e, to simplify the calculation.

The differential entropy yields a lower bound on the expected squared error of an estimator. For any random variable ${\displaystyle X}$ and estimator ${\displaystyle {\widehat {X}}}$ the following holds:[2]

${\displaystyle \operatorname {E} [(X-{\widehat {X}})^{2}]\geq {\frac {1}{2\pi e}}e^{2h(X)}}$

with equality if and only if ${\displaystyle X}$ is a Gaussian random variable and ${\displaystyle {\widehat {X}}}$ is the mean of ${\displaystyle X}$.

In the table below ${\displaystyle \Gamma (x)=\int _{0}^{\infty }e^{-t}t^{x-1}dt}$ is the gamma function, ${\displaystyle \psi (x)={\frac {d}{dx}}\ln \Gamma (x)={\frac {\Gamma '(x)}{\Gamma (x)}}}$ is the digamma function, ${\displaystyle B(p,q)={\frac {\Gamma (p)\Gamma (q)}{\Gamma (p+q)}}}$ is the beta function, and γ_E is Euler's constant.[8]^{: 219–230}

Table of differential entropies

Distribution Name	Probability density function (pdf)	Differential entropy in nats	Support
Uniform	${\displaystyle f(x)={\frac {1}{b-a}}}$	${\displaystyle \ln(b-a)\,}$	${\displaystyle [a,b]\,}$
Normal	${\displaystyle f(x)={\frac {1}{\sqrt {2\pi \sigma ^{2}}}}\exp \left(-{\frac {(x-\mu )^{2}}{2\sigma ^{2}}}\right)}$	${\displaystyle \ln \left(\sigma {\sqrt {2\,\pi \,e}}\right)}$	${\displaystyle (-\infty ,\infty )\,}$
Exponential	${\displaystyle f(x)=\lambda \exp \left(-\lambda x\right)}$	${\displaystyle 1-\ln \lambda \,}$	${\displaystyle [0,\infty )\,}$
Rayleigh	${\displaystyle f(x)={\frac {x}{\sigma ^{2}}}\exp \left(-{\frac {x^{2}}{2\sigma ^{2}}}\right)}$	${\displaystyle 1+\ln {\frac {\sigma }{\sqrt {2}}}+{\frac {\gamma _{E}}{2}}}$	${\displaystyle [0,\infty )\,}$
Beta	${\displaystyle f(x)={\frac {x^{\alpha -1}(1-x)^{\beta -1}}{B(\alpha ,\beta )}}}$ for ${\displaystyle 0\leq x\leq 1}$	${\displaystyle \ln B(\alpha ,\beta )-(\alpha -1)[\psi (\alpha )-\psi (\alpha +\beta )]\,}$ ${\displaystyle -(\beta -1)[\psi (\beta )-\psi (\alpha +\beta )]\,}$	${\displaystyle [0,1]\,}$
Cauchy	${\displaystyle f(x)={\frac {\gamma }{\pi }}{\frac {1}{\gamma ^{2}+x^{2}}}}$	${\displaystyle \ln(4\pi \gamma )\,}$	${\displaystyle (-\infty ,\infty )\,}$
Chi	${\displaystyle f(x)={\frac {2}{2^{k/2}\Gamma (k/2)}}x^{k-1}\exp \left(-{\frac {x^{2}}{2}}\right)}$	${\displaystyle \ln {\frac {\Gamma (k/2)}{\sqrt {2}}}-{\frac {k-1}{2}}\psi \left({\frac {k}{2}}\right)+{\frac {k}{2}}}$	${\displaystyle [0,\infty )\,}$
Chi-squared	${\displaystyle f(x)={\frac {1}{2^{k/2}\Gamma (k/2)}}x^{{\frac {k}{2}}\!-\!1}\exp \left(-{\frac {x}{2}}\right)}$	${\displaystyle \ln 2\Gamma \left({\frac {k}{2}}\right)-\left(1-{\frac {k}{2}}\right)\psi \left({\frac {k}{2}}\right)+{\frac {k}{2}}}$	${\displaystyle [0,\infty )\,}$
Erlang	${\displaystyle f(x)={\frac {\lambda ^{k}}{(k-1)!}}x^{k-1}\exp(-\lambda x)}$	${\displaystyle (1-k)\psi (k)+\ln {\frac {\Gamma (k)}{\lambda }}+k}$	${\displaystyle [0,\infty )\,}$
F	${\displaystyle f(x)={\frac {n_{1}^{\frac {n_{1}}{2}}n_{2}^{\frac {n_{2}}{2}}}{B({\frac {n_{1}}{2}},{\frac {n_{2}}{2}})}}{\frac {x^{{\frac {n_{1}}{2}}-1}}{(n_{2}+n_{1}x)^{\frac {n_{1}+n2}{2}}}}}$	${\displaystyle \ln {\frac {n_{1}}{n_{2}}}B\left({\frac {n_{1}}{2}},{\frac {n_{2}}{2}}\right)+\left(1-{\frac {n_{1}}{2}}\right)\psi \left({\frac {n_{1}}{2}}\right)-}$ ${\displaystyle \left(1+{\frac {n_{2}}{2}}\right)\psi \left({\frac {n_{2}}{2}}\right)+{\frac {n_{1}+n_{2}}{2}}\psi \left({\frac {n_{1}\!+\!n_{2}}{2}}\right)}$	${\displaystyle [0,\infty )\,}$
Gamma	${\displaystyle f(x)={\frac {x^{k-1}\exp(-{\frac {x}{\theta }})}{\theta ^{k}\Gamma (k)}}}$	${\displaystyle \ln(\theta \Gamma (k))+(1-k)\psi (k)+k\,}$	${\displaystyle [0,\infty )\,}$
Laplace	${\displaystyle f(x)={\frac {1}{2b}}\exp \left(-{\frac {|x-\mu |}{b}}\right)}$	${\displaystyle 1+\ln(2b)\,}$	${\displaystyle (-\infty ,\infty )\,}$
Logistic	${\displaystyle f(x)={\frac {e^{-x/s}}{s(1+e^{-x/s})^{2}}}}$	${\displaystyle \ln s+2\,}$	${\displaystyle (-\infty ,\infty )\,}$
Lognormal	${\displaystyle f(x)={\frac {1}{\sigma x{\sqrt {2\pi }}}}\exp \left(-{\frac {(\ln x-\mu )^{2}}{2\sigma ^{2}}}\right)}$	${\displaystyle \mu +{\frac {1}{2}}\ln(2\pi e\sigma ^{2})}$	${\displaystyle [0,\infty )\,}$
Maxwell–Boltzmann	${\displaystyle f(x)={\frac {1}{a^{3}}}{\sqrt {\frac {2}{\pi }}}\,x^{2}\exp \left(-{\frac {x^{2}}{2a^{2}}}\right)}$	${\displaystyle \ln(a{\sqrt {2\pi }})+\gamma _{E}-{\frac {1}{2}}}$	${\displaystyle [0,\infty )\,}$
Generalized normal	${\displaystyle f(x)={\frac {2\beta ^{\frac {\alpha }{2}}}{\Gamma ({\frac {\alpha }{2}})}}x^{\alpha -1}\exp(-\beta x^{2})}$	${\displaystyle \ln {\frac {\Gamma (\alpha /2)}{2\beta ^{\frac {1}{2}}}}-{\frac {\alpha -1}{2}}\psi \left({\frac {\alpha }{2}}\right)+{\frac {\alpha }{2}}}$	${\displaystyle (-\infty ,\infty )\,}$
Pareto	${\displaystyle f(x)={\frac {\alpha x_{m}^{\alpha }}{x^{\alpha +1}}}}$	${\displaystyle \ln {\frac {x_{m}}{\alpha }}+1+{\frac {1}{\alpha }}}$	${\displaystyle [x_{m},\infty )\,}$
Student's t	${\displaystyle f(x)={\frac {(1+x^{2}/\nu )^{-{\frac {\nu +1}{2}}}}{{\sqrt {\nu }}B({\frac {1}{2}},{\frac {\nu }{2}})}}}$	${\displaystyle {\frac {\nu \!+\!1}{2}}\left(\psi \left({\frac {\nu \!+\!1}{2}}\right)\!-\!\psi \left({\frac {\nu }{2}}\right)\right)\!+\!\ln {\sqrt {\nu }}B\left({\frac {1}{2}},{\frac {\nu }{2}}\right)}$	${\displaystyle (-\infty ,\infty )\,}$
Triangular	${\displaystyle f(x)={\begin{cases}{\frac {2(x-a)}{(b-a)(c-a)}}&\mathrm {for\ } a\leq x\leq c,\\[4pt]{\frac {2(b-x)}{(b-a)(b-c)}}&\mathrm {for\ } c<x\leq b,\\[4pt]\end{cases}}}$	${\displaystyle {\frac {1}{2}}+\ln {\frac {b-a}{2}}}$	${\displaystyle [a,b]\,}$
Weibull	${\displaystyle f(x)={\frac {k}{\lambda ^{k}}}x^{k-1}\exp \left(-{\frac {x^{k}}{\lambda ^{k}}}\right)}$	${\displaystyle {\frac {(k-1)\gamma _{E}}{k}}+\ln {\frac {\lambda }{k}}+1}$	${\displaystyle [0,\infty )\,}$
Multivariate normal	${\displaystyle f_{X}({\vec {x}})=}$ ${\displaystyle {\frac {\exp \left(-{\frac {1}{2}}({\vec {x}}-{\vec {\mu }})^{\top }\Sigma ^{-1}\cdot ({\vec {x}}-{\vec {\mu }})\right)}{(2\pi )^{N/2}\left|\Sigma \right|^{1/2}}}}$	${\displaystyle {\frac {1}{2}}\ln\{(2\pi e)^{N}\det(\Sigma )\}}$	${\displaystyle \mathbb {R} ^{N}}$

Many of the differential entropies are from.[9]^{: 120–122}

As described above, differential entropy does not share all properties of discrete entropy. For example, the differential entropy can be negative; also it is not invariant under continuous coordinate transformations. Edwin Thompson Jaynes showed in fact that the expression above is not the correct limit of the expression for a finite set of probabilities.[10]^{: 181–218}

A modification of differential entropy adds an invariant measure factor to correct this, (see limiting density of discrete points). If ${\displaystyle m(x)}$ is further constrained to be a probability density, the resulting notion is called relative entropy in information theory:

${\displaystyle D(p||m)=\int p(x)\log {\frac {p(x)}{m(x)}}\,dx.}$

The definition of differential entropy above can be obtained by partitioning the range of ${\displaystyle X}$ into bins of length ${\displaystyle h}$ with associated sample points ${\displaystyle ih}$ within the bins, for ${\displaystyle X}$ Riemann integrable. This gives a quantized version of ${\displaystyle X}$, defined by ${\displaystyle X_{h}=ih}$ if ${\displaystyle ih\leq X\leq (i+1)h}$. Then the entropy of ${\displaystyle X_{h}=ih}$ is[2]

${\displaystyle H_{h}=-\sum _{i}hf(ih)\log(f(ih))-\sum hf(ih)\log(h).}$

The first term on the right approximates the differential entropy, while the second term is approximately ${\displaystyle -\log(h)}$. Note that this procedure suggests that the entropy in the discrete sense of a continuous random variable should be ${\displaystyle \infty }$.

• Information entropy
• Self-information
• Entropy estimation

• "Differential entropy", Encyclopedia of Mathematics, EMS Press, 2001 [1994]
• "Differential entropy". PlanetMath.