Personal tools
You are here: Home → Electrical and Computer Engineering → Information Theory → Definitions and Basic Facts

Definitions and Basic Facts

Document Actions
  • Send this
  • Print this
  • Content View
  • Bookmarks
  • CourseFeed

Entropy Function   ::   Joint Entropy   ::   Relative Entropy   ::   Multivariable   ::   Convexity

More than two variables

It will be important to deal with more than one or two variables, and a variety of "chain rules" have been developed for this purpose. In each of these, the sequence of r.v.s $X_1,X_2,\ldots, X_n$ are drawn according to the joint distribution $p(x_1,x_2,\ldots,x_n)$.

\begin{theorem}The joint entropy of $X_1,X_2,\ldots, X_n$\ is \begin{displaymath... ...n) = \sum_{i=1}^n H(X_i\vert X_{i-1},\ldots,X_1). \end{displaymath}\end{theorem}


\begin{proof} Observe that \begin{displaymath}\begin{aligned} H(X_1,X_2) &= H(X... ...vert x_{i-1},\ldots,x_1) \end{displaymath}and taking an expectation. \end{proof}

We sometimes have two variables that we wish to consider, both conditioned upon another variable.

\begin{definition} The {\bf conditional mutual information} of random variables... ...\vert Z)}{p(X\vert Z)p(Y\vert Z)} \end{aligned}\end{displaymath}\end{definition}
In other words, it is the same as mutual information, but everything is conditioned upon Z.

The chain rule for entropy leads us to a chain rule for mutual information.

\begin{theorem} \begin{displaymath}I(X_1,X_2,\ldots,X_n;Y) = \sum_{i=1}^n I(X_i;Y\vert X_{i-1},X_{i-2},\ldots,X_{1}). \end{displaymath}\end{theorem}

\begin{proof} \begin{displaymath}\begin{aligned} I(X_1,X_2,\ldots,X_n;Y) &= H(X_... ...;Y\vert X_{i-1},X_{i-2},\ldots,X_{1}). \end{aligned}\end{displaymath}\end{proof}

(Skip conditional relative entropy for now.)

Copyright 2008, by the Contributing Authors. Cite/attribute Resource. admin. (2006, May 17). Definitions and Basic Facts. Retrieved November 24, 2009, from Free Online Course Materials — USU OpenCourseWare Web site: http://ocw.usu.edu/Electrical_and_Computer_Engineering/Information_Theory/lecture2_4.htm. This work is licensed under a Creative Commons License. Creative Commons License
Reuse Course
Download this course