<aside> 📌 By Dr. Nir Regev
</aside>
<aside> 📌 Sign up to Circuit of Knowledge blog for unlimited tutorials and content
</aside>
<aside> 📌 If it’s knowledge you’re after, join our growing Slack community!
</aside>
July 5th 2024
Jensen's inequality is a fundamental result in probability theory and statistics that relates the value of a convex function of an expectation to the expectation of the convex function. It states that for a convex function $f$ and a random variable $X$, the following inequality holds:
$$ f(\mathbb{E}[X]) \leq \mathbb{E}[f(X)] $$
In other words, the value of the convex function evaluated at the expected value of $X$ is less than or equal to the expected value of the convex function evaluated at $X$.
Let $f$ be a convex function and $X$ be a random variable with probability density function $p(x)$. The expectation of $X$ is given by:
$$ \mathbb{E}[X] = \int x \, p(x) \, dx $$
Jensen's inequality states that:
$$ f(\mathbb{E}[X]) \leq \mathbb{E}[f(X)] $$
which can be written as:
$$ f\left(\int x \, p(x) \, dx\right) \leq \int f(x) \, p(x) \, dx $$
The inequality holds for any convex function $f$. A function $f$ is said to be convex if for any two points $x_1$ and $x_2$ in its domain and any $\lambda \in [0, 1]$, the following inequality holds:
$$ f(\lambda x_1 + (1 - \lambda) x_2) \leq \lambda f(x_1) + (1 - \lambda) f(x_2) $$
Intuitively, this means that the line segment connecting any two points on the graph of a convex function lies above or on the graph.
Let's demonstrate Jensen's inequality using Python code. We'll use the exponential function, which is a convex function, as an example.