Personal tools
  •  
You are here: Home Electrical and Computer Engineering Information Theory Definitions and Basic Facts

Definitions and Basic Facts

Document Actions
  • Content View
  • Bookmarks
  • CourseFeed

Entropy Function   ::   Joint Entropy   ::   Relative Entropy   ::   Multivariable   ::   Convexity

More than two variables

It will be important to deal with more than one or two variables, and a variety of "chain rules" have been developed for this purpose. In each of these, the sequence of r.v.s $X_1,X_2,\ldots, X_n$ are drawn according to the joint distribution $p(x_1,x_2,\ldots,x_n)$ .

\begin{theorem}The joint entropy of $X_1,X_2,\ldots, X_n$\ is
\begin{displaymath...
...n) = \sum_{i=1}^n H(X_i\vert X_{i-1},\ldots,X_1).
\end{displaymath}\end{theorem}


\begin{proof}
Observe that
\begin{displaymath}\begin{aligned}
H(X_1,X_2) &= H(X...
...vert x_{i-1},\ldots,x_1)
\end{displaymath}and taking an expectation.
\end{proof}

We sometimes have two variables that we wish to consider, both conditioned upon another variable.

\begin{definition}
The {\bf conditional mutual information} of random variables...
...\vert Z)}{p(X\vert Z)p(Y\vert Z)}
\end{aligned}\end{displaymath}\end{definition}
In other words, it is the same as mutual information, but everything is conditioned upon Z .

The chain rule for entropy leads us to a chain rule for mutual information .

\begin{theorem}
\begin{displaymath}I(X_1,X_2,\ldots,X_n;Y) = \sum_{i=1}^n
I(X_i;Y\vert X_{i-1},X_{i-2},\ldots,X_{1}).
\end{displaymath}\end{theorem}

\begin{proof}
\begin{displaymath}\begin{aligned}
I(X_1,X_2,\ldots,X_n;Y) &= H(X_...
...;Y\vert X_{i-1},X_{i-2},\ldots,X_{1}).
\end{aligned}\end{displaymath}\end{proof}

(Skip conditional relative entropy for now.)

Copyright 2008, by the Contributing Authors. Cite/attribute Resource . admin. (2006, May 17). Definitions and Basic Facts. Retrieved January 07, 2011, from Free Online Course Materials — USU OpenCourseWare Web site: http://ocw.usu.edu/Electrical_and_Computer_Engineering/Information_Theory/lecture2_4.htm. This work is licensed under a Creative Commons License Creative Commons License