Jensen's inequality

For a real convex function ${\displaystyle \varphi }$, numbers ${\displaystyle x_{1},x_{2},\ldots ,x_{n}}$ in its domain, and positive weights ${\displaystyle a_{i}}$, Jensen's inequality can be stated as:

${\displaystyle \varphi \left({\frac {\sum a_{i}x_{i}}{\sum a_{i}}}\right)\leq {\frac {\sum a_{i}\varphi (x_{i})}{\sum a_{i}}}}$

(1)

and the inequality is reversed if ${\displaystyle \varphi }$ is concave, which is

${\displaystyle \varphi \left({\frac {\sum a_{i}x_{i}}{\sum a_{i}}}\right)\geq {\frac {\sum a_{i}\varphi (x_{i})}{\sum a_{i}}}.}$

(2)

Equality holds if and only if ${\displaystyle x_{1}=x_{2}=\cdots =x_{n}}$ or ${\displaystyle \varphi }$ is linear on a domain containing ${\displaystyle x_{1},x_{2},\cdots ,x_{n}}$.

As a particular case, if the weights ${\displaystyle a_{i}}$ are all equal, then (1) and (2) become

${\displaystyle \varphi \left({\frac {\sum x_{i}}{n}}\right)\leq {\frac {\sum \varphi (x_{i})}{n}}}$

(3)

${\displaystyle \varphi \left({\frac {\sum x_{i}}{n}}\right)\geq {\frac {\sum \varphi (x_{i})}{n}}}$

(4)

For instance, the function log(x) is concave, so substituting ${\displaystyle \varphi (x)=\log(x)}$ in the previous formula (4) establishes the (logarithm of the) familiar arithmetic-mean/geometric-mean inequality:

${\displaystyle \log \!\left({\frac {\sum _{i=1}^{n}x_{i}}{n}}\right)\geq {\frac {\sum _{i=1}^{n}\log \!\left(x_{i}\right)}{n}}\quad {\text{or}}\quad {\frac {x_{1}+x_{2}+\cdots +x_{n}}{n}}\geq {\sqrt[{n}]{x_{1}\cdot x_{2}\cdots x_{n}}}}$

A common application has x as a function of another variable (or set of variables) t, that is, ${\displaystyle x_{i}=g(t_{i})}$. All of this carries directly over to the general continuous case: the weights a_i are replaced by a non-negative integrable function  f (x), such as a probability distribution, and the summations are replaced by integrals.

Let ${\displaystyle (\Omega ,A,\mu )}$ be a probability space. Let ${\displaystyle f:\Omega \to \mathbb {R} }$ be a ${\displaystyle \mu }$-measurable function and ${\displaystyle \varphi$ :\mathbb {R} \to \mathbb {R} } be convex. Then:[5]

$${\displaystyle \varphi \left(\int _{\Omega }f\,\mathrm {d} \mu \right)\leq \int _{\Omega }\varphi \circ f\,\mathrm {d} \mu }$$

In real analysis, we may require an estimate on

${\displaystyle \varphi \left(\int _{a}^{b}f(x)\,dx\right)}$

where ${\displaystyle a,b\in \mathbb {R} }$, and ${\displaystyle f\colon [a,b]\to \mathbb {R} }$ is a non-negative Lebesgue-integrable function. In this case, the Lebesgue measure of ${\displaystyle [a,b]}$ need not be unity. However, by integration by substitution, the interval can be rescaled so that it has measure unity. Then Jensen's inequality can be applied to get[6]

${\displaystyle \varphi \left({\frac {1}{b-a}}\int _{a}^{b}f(x)\,dx\right)\leq {\frac {1}{b-a}}\int _{a}^{b}\varphi (f(x))\,dx.}$

The same result can be equivalently stated in a probability theory setting, by a simple change of notation. Let ${\displaystyle (\Omega ,{\mathfrak {F}},\operatorname {P} )}$ be a probability space, X an integrable real-valued random variable and φ a convex function. Then:

${\displaystyle \varphi \left(\operatorname {E} [X]\right)\leq \operatorname {E} \left[\varphi (X)\right].}$[7]

In this probability setting, the measure μ is intended as a probability ${\displaystyle \operatorname {P} }$, the integral with respect to μ as an expected value ${\displaystyle \operatorname {E} }$, and the function ${\displaystyle f}$ as a random variable X.

Note that the equality holds if and only if φ is a linear function on some convex set ${\displaystyle A}$ such that ${\displaystyle \mathrm {P} (X\in A)=1}$ (which follows by inspecting the measure-theoretical proof below).

More generally, let T be a real topological vector space, and X a T-valued integrable random variable. In this general setting, integrable means that there exists an element ${\displaystyle \operatorname {E} [X]}$ in T, such that for any element z in the dual space of T: ${\displaystyle \operatorname {E} |\langle z,X\rangle |<\infty }$, and ${\displaystyle \langle z,\operatorname {E} [X]\rangle =\operatorname {E} [\langle z,X\rangle ]}$. Then, for any measurable convex function φ and any sub-σ-algebra ${\displaystyle {\mathfrak {G}}}$ of ${\displaystyle {\mathfrak {F}}}$:

${\displaystyle \varphi \left(\operatorname {E} \left[X\mid {\mathfrak {G}}\right]\right)\leq \operatorname {E} \left[\varphi (X)\mid {\mathfrak {G}}\right].}$

Here ${\displaystyle \operatorname {E} [\cdot \mid {\mathfrak {G}}]}$ stands for the expectation conditioned to the σ-algebra ${\displaystyle {\mathfrak {G}}}$. This general statement reduces to the previous ones when the topological vector space T is the real axis, and ${\displaystyle {\mathfrak {G}}}$ is the trivial σ-algebra {∅, Ω} (where ∅ is the empty set, and Ω is the sample space).[8]

Let X be a one-dimensional random variable with mean ${\displaystyle \mu }$ and variance ${\displaystyle \sigma ^{2}\geq 0}$. Let ${\displaystyle \varphi (x)}$ be a twice differentiable function, and define the function

${\displaystyle h(x)\triangleq {\frac {\varphi \left(x\right)-\varphi \left(\mu \right)}{\left(x-\mu \right)^{2}}}-{\frac {\varphi '\left(\mu \right)}{x-\mu }}.}$

Then[9]

${\displaystyle \sigma ^{2}\inf {\frac {\varphi ''(x)}{2}}\leq \sigma ^{2}\inf h(x)\leq E\left[\varphi \left(X\right)\right]-\varphi \left(E[X]\right)\leq \sigma ^{2}\sup h(x)\leq \sigma ^{2}\sup {\frac {\varphi ''(x)}{2}}.}$

In particular, when ${\displaystyle \varphi (x)}$ is convex, then ${\displaystyle \varphi ''(x)\geq 0}$, and the standard form of Jensen's inequality immediately follows for the case where ${\displaystyle \varphi (x)}$ is additionally assumed to be twice differentiable.

If λ₁ and λ₂ are two arbitrary nonnegative real numbers such that λ₁ + λ₂ = 1 then convexity of φ implies

${\displaystyle \forall x_{1},x_{2}:\qquad \varphi \left(\lambda _{1}x_{1}+\lambda _{2}x_{2}\right)\leq \lambda _{1}\,\varphi (x_{1})+\lambda _{2}\,\varphi (x_{2}).}$

This can be generalized: if λ₁, ..., λ_n are nonnegative real numbers such that λ₁ + ... + λ_n = 1, then

${\displaystyle \varphi (\lambda _{1}x_{1}+\lambda _{2}x_{2}+\cdots +\lambda _{n}x_{n})\leq \lambda _{1}\,\varphi (x_{1})+\lambda _{2}\,\varphi (x_{2})+\cdots +\lambda _{n}\,\varphi (x_{n}),}$

for any x₁, ..., x_n.

The finite form of the Jensen's inequality can be proved by induction: by convexity hypotheses, the statement is true for n = 2. Suppose the statement is true for some n, so

${\displaystyle \varphi \left(\sum _{i=1}^{n}\lambda _{i}x_{i}\right)\leq \sum _{i=1}^{n}\lambda _{i}\varphi \left(x_{i}\right)}$

for any λ₁, ..., λ_n such that λ₁ + ... + λ_n = 1.

One needs to prove it for n + 1. At least one of the λ_i is strictly smaller than ${\displaystyle 1}$, say λ_n+1; therefore by convexity inequality:

${\displaystyle {\begin{aligned}\varphi \left(\sum _{i=1}^{n+1}\lambda _{i}x_{i}\right)&=\varphi \left((1-\lambda _{n+1})\sum _{i=1}^{n}{\frac {\lambda _{i}}{1-\lambda _{n+1}}}x_{i}+\lambda _{n+1}x_{n+1}\right)\\&\leq (1-\lambda _{n+1})\varphi \left(\sum _{i=1}^{n}{\frac {\lambda _{i}}{1-\lambda _{n+1}}}x_{i}\right)+\lambda _{n+1}\,\varphi (x_{n+1}).\end{aligned}}}$

Since λ₁ + ... +λ_n + λ_n+1 = 1,

${\displaystyle \sum _{i=1}^{n}{\frac {\lambda _{i}}{1-\lambda _{n+1}}}=1}$,

applying the inductive hypothesis gives

${\displaystyle \varphi \left(\sum _{i=1}^{n}{\frac {\lambda _{i}}{1-\lambda _{n+1}}}x_{i}\right)\leq \sum _{i=1}^{n}{\frac {\lambda _{i}}{1-\lambda _{n+1}}}\varphi (x_{i})}$

therefore

${\displaystyle {\begin{aligned}\varphi \left(\sum _{i=1}^{n+1}\lambda _{i}x_{i}\right)&\leq (1-\lambda _{n+1})\sum _{i=1}^{n}{\frac {\lambda _{i}}{1-\lambda _{n+1}}}\varphi (x_{i})+\lambda _{n+1}\,\varphi (x_{n+1})=\sum _{i=1}^{n+1}\lambda _{i}\varphi (x_{i})\end{aligned}}}$

We deduce the equality is true for n + 1, by induction it follows that the result is also true for all integer n greater than 2.

In order to obtain the general inequality from this finite form, one needs to use a density argument. The finite form can be rewritten as:

${\displaystyle \varphi \left(\int x\,d\mu _{n}(x)\right)\leq \int \varphi (x)\,d\mu _{n}(x),}$

where μ_n is a measure given by an arbitrary convex combination of Dirac deltas:

${\displaystyle \mu _{n}=\sum _{i=1}^{n}\lambda _{i}\delta _{x_{i}}.}$

Since convex functions are continuous, and since convex combinations of Dirac deltas are weakly dense in the set of probability measures (as could be easily verified), the general statement is obtained simply by a limiting procedure.

Let ${\displaystyle g}$ be a real-valued ${\displaystyle \mu }$-integrable function on a probability space ${\displaystyle \Omega }$, and let ${\displaystyle \varphi }$ be a convex function on the real numbers. Since ${\displaystyle \varphi }$ is convex, at each real number ${\displaystyle x}$ we have a nonempty set of subderivatives, which may be thought of as lines touching the graph of ${\displaystyle \varphi }$ at ${\displaystyle x}$, but which are below the graph of ${\displaystyle \varphi }$ at all points (support lines of the graph).

Now, if we define

${\displaystyle x_{0}:=\int _{\Omega }g\,d\mu ,}$

because of the existence of subderivatives for convex functions, we may choose ${\displaystyle a}$ and ${\displaystyle b}$ such that

${\displaystyle ax+b\leq \varphi (x),}$

for all real ${\displaystyle x}$ and

${\displaystyle ax_{0}+b=\varphi (x_{0}).}$

But then we have that

${\displaystyle \varphi \circ g(\omega )\geq ag(\omega )+b}$

for almost all ${\displaystyle \omega \in \Omega }$. Since we have a probability measure, the integral is monotone with ${\displaystyle \mu (\Omega )=1}$ so that

${\displaystyle \int _{\Omega }\varphi \circ g\,d\mu \geq \int _{\Omega }(ag+b)\,d\mu =a\int _{\Omega }g\,d\mu +b\int _{\Omega }d\mu =ax_{0}+b=\varphi (x_{0})=\varphi \left(\int _{\Omega }g\,d\mu \right),}$

as desired.

Let X be an integrable random variable that takes values in a real topological vector space T. Since ${\displaystyle \varphi :T\to \mathbb {R} }$ is convex, for any ${\displaystyle x,y\in T}$, the quantity

${\displaystyle {\frac {\varphi (x+\theta \,y)-\varphi (x)}{\theta }},}$

is decreasing as θ approaches 0⁺. In particular, the subdifferential of ${\displaystyle \varphi }$ evaluated at x in the direction y is well-defined by

${\displaystyle (D\varphi )(x)\cdot y:=\lim _{\theta \downarrow 0}{\frac {\varphi (x+\theta \,y)-\varphi (x)}{\theta }}=\inf _{\theta \neq 0}{\frac {\varphi (x+\theta \,y)-\varphi (x)}{\theta }}.}$

It is easily seen that the subdifferential is linear in y (that is false and the assertion requires Hahn-Banach theorem to be proved) and, since the infimum taken in the right-hand side of the previous formula is smaller than the value of the same term for θ = 1, one gets

${\displaystyle \varphi (x)\leq \varphi (x+y)-(D\varphi )(x)\cdot y.}$

In particular, for an arbitrary sub-σ-algebra ${\displaystyle {\mathfrak {G}}}$ we can evaluate the last inequality when ${\displaystyle x=\operatorname {E} [X\mid {\mathfrak {G}}],\,y=X-\operatorname {E} [X\mid {\mathfrak {G}}]}$ to obtain

${\displaystyle \varphi (\operatorname {E} [X\mid {\mathfrak {G}}])\leq \varphi (X)-(D\varphi )(\operatorname {E} [X\mid {\mathfrak {G}}])\cdot (X-\operatorname {E} [X\mid {\mathfrak {G}}]).}$

Now, if we take the expectation conditioned to ${\displaystyle {\mathfrak {G}}}$ on both sides of the previous expression, we get the result since:

${\displaystyle \operatorname {E} \left[\left[(D\varphi )(\operatorname {E} [X\mid {\mathfrak {G}}])\cdot (X-\operatorname {E} [X\mid {\mathfrak {G}}])\right]\mid {\mathfrak {G}}\right]=(D\varphi )(\operatorname {E} [X\mid {\mathfrak {G}}])\cdot \operatorname {E} [\left(X-\operatorname {E} [X\mid {\mathfrak {G}}]\right)\mid {\mathfrak {G}}]=0,}$

by the linearity of the subdifferential in the y variable, and the following well-known property of the conditional expectation:

${\displaystyle \operatorname {E} \left[\left(\operatorname {E} [X\mid {\mathfrak {G}}]\right)\mid {\mathfrak {G}}\right]=\operatorname {E} [X\mid {\mathfrak {G}}].}$

Suppose Ω is a measurable subset of the real line and f(x) is a non-negative function such that

${\displaystyle \int _{-\infty }^{\infty }f(x)\,dx=1.}$

In probabilistic language, f is a probability density function.

Then Jensen's inequality becomes the following statement about convex integrals:

If g is any real-valued measurable function and ${\textstyle \varphi }$ is convex over the range of g, then

${\displaystyle \varphi \left(\int _{-\infty }^{\infty }g(x)f(x)\,dx\right)\leq \int _{-\infty }^{\infty }\varphi (g(x))f(x)\,dx.}$

If g(x) = x, then this form of the inequality reduces to a commonly used special case:

${\displaystyle \varphi \left(\int _{-\infty }^{\infty }x\,f(x)\,dx\right)\leq \int _{-\infty }^{\infty }\varphi (x)\,f(x)\,dx.}$

This is applied in Variational Bayesian methods.

If g(x) = x²ⁿ, and X is a random variable, then g is convex as

${\displaystyle {\frac {d^{2}g}{dx^{2}}}(x)=2n(2n-1)x^{2n-2}\geq 0\quad \forall \ x\in \mathbb {R} }$

and so

${\displaystyle g(\operatorname {E} [X])=(\operatorname {E} [X])^{2n}\leq \operatorname {E} [X^{2n}].}$

In particular, if some even moment 2n of X is finite, X has a finite mean. An extension of this argument shows X has finite moments of every order ${\displaystyle l\in \mathbb {N} }$ dividing n.

Let Ω = {x₁, ... x_n}, and take μ to be the counting measure on Ω, then the general form reduces to a statement about sums:

${\displaystyle \varphi \left(\sum _{i=1}^{n}g(x_{i})\lambda _{i}\right)\leq \sum _{i=1}^{n}\varphi (g(x_{i}))\lambda _{i},}$

provided that λ_i ≥ 0 and

${\displaystyle \lambda _{1}+\cdots +\lambda _{n}=1.}$

There is also an infinite discrete form.

Jensen's inequality is of particular importance in statistical physics when the convex function is an exponential, giving:

${\displaystyle e^{\operatorname {E} [X]}\leq \operatorname {E} \left[e^{X}\right],}$

where the expected values are with respect to some probability distribution in the random variable X.

Proof: Let ${\displaystyle \varphi (x)=e^{x}}$ in ${\displaystyle \varphi \left(\operatorname {E} [X]\right)\leq \operatorname {E} \left[\varphi (X)\right].}$

If p(x) is the true probability density for X, and q(x) is another density, then applying Jensen's inequality for the random variable Y(X) = q(X)/p(X) and the convex function φ(y) = −log(y) gives

${\displaystyle \operatorname {E} [\varphi (Y)]\geq \varphi (\operatorname {E} [Y])}$

Therefore:

${\displaystyle -D(p(x)\|q(x))=\int p(x)\log \left({\frac {q(x)}{p(x)}}\right)\,dx\leq \log \left(\int p(x){\frac {q(x)}{p(x)}}\,dx\right)=\log \left(\int q(x)\,dx\right)=0}$

a result called Gibbs' inequality.

It shows that the average message length is minimised when codes are assigned on the basis of the true probabilities p rather than any other distribution q. The quantity that is non-negative is called the Kullback–Leibler divergence of q from p.

Since −log(x) is a strictly convex function for x > 0, it follows that equality holds when p(x) equals q(x) almost everywhere.

If L is a convex function and ${\displaystyle {\mathfrak {G}}}$ a sub-sigma-algebra, then, from the conditional version of Jensen's inequality, we get

${\displaystyle L(\operatorname {E} [\delta (X)\mid {\mathfrak {G}}])\leq \operatorname {E} [L(\delta (X))\mid {\mathfrak {G}}]\quad \Longrightarrow \quad \operatorname {E} [L(\operatorname {E} [\delta (X)\mid {\mathfrak {G}}])]\leq \operatorname {E} [L(\delta (X))].}$

So if δ(X) is some estimator of an unobserved parameter θ given a vector of observables X; and if T(X) is a sufficient statistic for θ; then an improved estimator, in the sense of having a smaller expected loss L, can be obtained by calculating

${\displaystyle \delta _{1}(X)=\operatorname {E} _{\theta }[\delta (X')\mid T(X')=T(X)],}$

the expected value of δ with respect to θ, taken over all possible vectors of observations X compatible with the same value of T(X) as that observed. Further, because T is a sufficient statistics, ${\displaystyle \delta _{1}(X)}$ does not depend on θ, hence, becomes a statistics.

This result is known as the Rao–Blackwell theorem.

A popular method of measuring the investment performance of an investment is the Internal Rate of Return (IRR) which is the rate by which a series of uncertain future cash flows are discounted using Present Value Theory to cause the sum of the future cash flows to equal the initial investment. While it is tempting to perform Monte Carlo simulation of the IRR, Jensen's Inequality introduces a bias due to fact that the IRR function is a curved function and the expectation operator is a linear function.[11]