Some More Bounds

Document Actions

Log Sum Inequality :: Data Processing Inequality :: Fano's Inequality

Fano's Inequality

A fundamental operation in communications is estimating a value based on some measurement. That is, suppose a value X is sent through a channel (where it is corrupted) and a value Y is received. Based on that received value we want to determine an estimate of X by performing some function on the observed value Y. Denote the estimate of X by $\Xhat$ :

$\begin{displaymath}\Xhat = g(Y).\end{displaymath}$

A question of performance now arises naturally: what is the probability that we have estimated the correct value of X. This can be explored in a variety of ways. One of the ways that will be fruitful to us in this class is by Fano's inequality, which relates the probability of error to the conditional entropy H(X|Y). Intuitively, if there is little uncertainty about X when we know Y, then the probability of error should be small. In fact, when H(X|Y)=0, then the probability of error should be zero: there is no uncertainty left over after we observe Y. Fano's inequality makes a quantitative statement to this effect.

Let

$\begin{displaymath}P_e = \text{probability of error} = \text{Pr}\{\Xhat \neq X\}. \end{displaymath}$

$\begin{theorem} (Fano's inequality) \begin{displaymath}\boxed{H(P_e) +P_e \log(... ...ymath}1+P_e \log(\vert\Xc\vert) \geq H(X\vert Y). \end{displaymath}\end{theorem}$

Note that if P_e=0 then H(X|Y)=0.

$\begin{proof} Define the random variable $E$\ by \begin{displaymath}E = \begin{... ...xt{(how many ways to make an error?)} \end{aligned}\end{displaymath}\end{proof}$

Copyright 2008, by the Contributing Authors. Cite/attribute Resource. admin. (2006, May 17). Some More Bounds. Retrieved August 30, 2008, from Free Online Course Materials — USU OpenCourseWare Web site: http://ocw.usu.edu/Electrical_and_Computer_Engineering/Information_Theory/lecture3_3.htm. This work is licensed under a Creative Commons License.

Course Contents Information Theory Home About the Professor Syllabus Schedule Homework Assignments Programming Assignments Download This Course	Personal tools Log in You are here: Home → Electrical and Computer Engineering → Information Theory → Some More Bounds
	Info Some More Bounds Document Actions Log Sum Inequality :: Data Processing Inequality :: Fano's Inequality Fano's Inequality A fundamental operation in communications is estimating a value based on some measurement. That is, suppose a value X is sent through a channel (where it is corrupted) and a value Y is received. Based on that received value we want to determine an estimate of X by performing some function on the observed value Y. Denote the estimate of X by $\Xhat$ : $\begin{displaymath}\Xhat = g(Y).\end{displaymath}$ A question of performance now arises naturally: what is the probability that we have estimated the correct value of X. This can be explored in a variety of ways. One of the ways that will be fruitful to us in this class is by Fano's inequality, which relates the probability of error to the conditional entropy H(X\|Y). Intuitively, if there is little uncertainty about X when we know Y, then the probability of error should be small. In fact, when H(X\|Y)=0, then the probability of error should be zero: there is no uncertainty left over after we observe Y. Fano's inequality makes a quantitative statement to this effect. Let $\begin{displaymath}P_e = \text{probability of error} = \text{Pr}\{\Xhat \neq X\}. \end{displaymath}$ $\begin{theorem} (Fano's inequality) \begin{displaymath}\boxed{H(P_e) +P_e \log(... ...ymath}1+P_e \log(\vert\Xc\vert) \geq H(X\vert Y). \end{displaymath}\end{theorem}$ Note that if P_e=0 then H(X\|Y)=0. $\begin{proof} Define the random variable $E$\ by \begin{displaymath}E = \begin{... ...xt{(how many ways to make an error?)} \end{aligned}\end{displaymath}\end{proof}$ Copyright 2008, by the Contributing Authors. Cite/attribute Resource. admin. (2006, May 17). Some More Bounds. Retrieved August 30, 2008, from Free Online Course Materials — USU OpenCourseWare Web site: http://ocw.usu.edu/Electrical_and_Computer_Engineering/Information_Theory/lecture3_3.htm. This work is licensed under a Creative Commons License.	Reuse Course Download this course

Sections

Personal tools

Some More Bounds

Document Actions

Fano's Inequality