Some More Bounds

Document Actions

Log Sum Inequality :: Data Processing Inequality :: Fano's Inequality

The Log Sum Inequality

In this section we introduce an inequality which will allow us to deduce the concavity (or convexity) of some many useful functions.

$\begin{theorem} (Log-sum inequality) For non-negative numbers $a_1,a_2,\ldots,a... ...1}^n b_i} \end{displaymath}with equality iff $a_i/b_i = constant$. \end{theorem}$

Note that the conditions on this theorem are much weaker than for Jensen's inequality, since it is not necessary to have the sets of numbers add up to 1.

$\begin{proof} The function $f(t) = t\log t$\ is strictly convex. (Why?). By Je... ...a_i}{\sum_j b_j} \log \sum_j \frac{a_i}{\sum_j b_j} \end{displaymath}\end{proof}$

Using this inequality, we can prove a convexity statement about the relative entropy function.

$\begin{theorem} If $(p_1,q_1)$\ and $(p_2,q_2)$\ are pairs of probability mass ... ...da \leq 1$. That is, $D(p\Vert q)$\ is convex in the pair $(p,q)$. \end{theorem}$

$\begin{proof} Recall that \begin{displaymath}D(p\Vert q) = \sum_x p(x) \log \fr... ...a D(p_1 \Vert q_2) + (1-\lambda) D(p_2 \Vert q_2). \end{multline}\par\end{proof}$

$\begin{theorem} $H(p)$\ is a concave function of $p$. \end{theorem}$

$\begin{proof} \begin{displaymath}H(p) = \log \vert\Xc\vert - D(p\Vert u) \end{di... ...he uniform distribution. Since $D$\ is convex, $H$\ must be concave. \end{proof}$

$\begin{proof} Here is another more direct proof. Let $X_1 \sim p_1$\ and $X_2 \... ...mbda)p_2) \geq \lambda H(p_1) + (1-\lambda) H(p_2). \end{displaymath}\end{proof}$

The following theorem is important and will be used several times throughout the quarter.

$\begin{theorem} Let $(X,Y) \sim p(x,y) = p(x)p(y\vert x)$. The mutual informati... ...rt x)$\ and a convex function of $p(y\vert x)$\ for fixed $p(x)$. \end{theorem}$

$\begin{proof} Recall that \begin{displaymath}p(y) = \sum_x p(x,y) = \sum_x p(x)... ...formation must be a convex function of the conditional distribution. \end{proof}$

Copyright 2008, Todd Moon. Cite/attribute Resource. admin. (2006, May 15). Some More Bounds. Retrieved November 24, 2009, from Free Online Course Materials — USU OpenCourseWare Web site: http://ocw.usu.edu/Electrical_and_Computer_Engineering/Information_Theory/lecture3_1.htm. This work is licensed under a Creative Commons License.

Course Contents Information Theory Home About the Professor Syllabus Schedule Homework Assignments Programming Assignments Download This Course	Personal tools Log in You are here: Home → Electrical and Computer Engineering → Information Theory → Some More Bounds
	Info Some More Bounds Document Actions Log Sum Inequality :: Data Processing Inequality :: Fano's Inequality The Log Sum Inequality In this section we introduce an inequality which will allow us to deduce the concavity (or convexity) of some many useful functions. $\begin{theorem} (Log-sum inequality) For non-negative numbers $a_1,a_2,\ldots,a... ...1}^n b_i} \end{displaymath}with equality iff $a_i/b_i = constant$. \end{theorem}$ Note that the conditions on this theorem are much weaker than for Jensen's inequality, since it is not necessary to have the sets of numbers add up to 1. $\begin{proof} The function $f(t) = t\log t$\ is strictly convex. (Why?). By Je... ...a_i}{\sum_j b_j} \log \sum_j \frac{a_i}{\sum_j b_j} \end{displaymath}\end{proof}$ Using this inequality, we can prove a convexity statement about the relative entropy function. $\begin{theorem} If $(p_1,q_1)$\ and $(p_2,q_2)$\ are pairs of probability mass ... ...da \leq 1$. That is, $D(p\Vert q)$\ is convex in the pair $(p,q)$. \end{theorem}$ $\begin{proof} Recall that \begin{displaymath}D(p\Vert q) = \sum_x p(x) \log \fr... ...a D(p_1 \Vert q_2) + (1-\lambda) D(p_2 \Vert q_2). \end{multline}\par\end{proof}$ $\begin{theorem} $H(p)$\ is a concave function of $p$. \end{theorem}$ $\begin{proof} \begin{displaymath}H(p) = \log \vert\Xc\vert - D(p\Vert u) \end{di... ...he uniform distribution. Since $D$\ is convex, $H$\ must be concave. \end{proof}$ $\begin{proof} Here is another more direct proof. Let $X_1 \sim p_1$\ and $X_2 \... ...mbda)p_2) \geq \lambda H(p_1) + (1-\lambda) H(p_2). \end{displaymath}\end{proof}$ The following theorem is important and will be used several times throughout the quarter. $\begin{theorem} Let $(X,Y) \sim p(x,y) = p(x)p(y\vert x)$. The mutual informati... ...rt x)$\ and a convex function of $p(y\vert x)$\ for fixed $p(x)$. \end{theorem}$ $\begin{proof} Recall that \begin{displaymath}p(y) = \sum_x p(x,y) = \sum_x p(x)... ...formation must be a convex function of the conditional distribution. \end{proof}$ Copyright 2008, Todd Moon. Cite/attribute Resource. admin. (2006, May 15). Some More Bounds. Retrieved November 24, 2009, from Free Online Course Materials — USU OpenCourseWare Web site: http://ocw.usu.edu/Electrical_and_Computer_Engineering/Information_Theory/lecture3_1.htm. This work is licensed under a Creative Commons License.	Reuse Course Download this course

Sections

Personal tools

Some More Bounds

Document Actions

The Log Sum Inequality