Personal tools
  •  
You are here: Home Electrical and Computer Engineering Information Theory The Asymptotic Equipartition Property

The Asymptotic Equipartition Property

Document Actions
  • Content View
  • Bookmarks
  • CourseFeed

Introduction   ::   Convergence   ::   Data Compression   ::   Sets

Introduction

The AEP is the weak law of large numbers as applied to the estimation of entropy.

The law of large numbers and AEP

The law of large numbers states that for i.i.d. random variables $X_1,X_2,\ldots,X_n$ the sum

\begin{displaymath}\frac{1}{n}\sum_{i=1}^n X_i
\end{displaymath}

is close to its expected value EX . (How close? How can we tell?) The AEP states that for i.i.d. r.v.'s,

\begin{displaymath}\frac{1}{n}\log \frac{1}{p(X_1,X_2,\ldots,X_n)}
\end{displaymath}

is close to H ( X ). To put it another way,

\begin{displaymath}p(x_1,x_2,\ldots,x_n) \approx 2^{-nH}
\end{displaymath}


\begin{example}
Suppose $X \in \{ \zerobf,\onebf\}$, with $p(\onebf) = p$,
$p(...
...e
probability. This is nothing more than the law of large numbers.
\end{example}


Copyright 2008, Todd Moon. Cite/attribute Resource . admin. (2006, May 15). The Asymptotic Equipartition Property. Retrieved January 07, 2011, from Free Online Course Materials — USU OpenCourseWare Web site: http://ocw.usu.edu/Electrical_and_Computer_Engineering/Information_Theory/equi.htm. This work is licensed under a Creative Commons License Creative Commons License