The Asymptotic Equipartition Property

Introduction :: Convergence :: Data Compression :: Sets

Introduction

The AEP is the weak law of large numbers as applied to the estimation of entropy.

The law of large numbers and AEP

The law of large numbers states that for i.i.d. random variables $X_1,X_2,\ldots,X_n$ the sum

$\begin{displaymath}\frac{1}{n}\sum_{i=1}^n X_i \end{displaymath}$

is close to its expected value EX. (How close? How can we tell?) The AEP states that for i.i.d. r.v.'s,

$\begin{displaymath}\frac{1}{n}\log \frac{1}{p(X_1,X_2,\ldots,X_n)} \end{displaymath}$

is close to H(X). To put it another way,

$\begin{displaymath}p(x_1,x_2,\ldots,x_n) \approx 2^{-nH} \end{displaymath}$

$\begin{example} Suppose $X \in \{ \zerobf,\onebf\}$, with $p(\onebf) = p$, $p(... ...e probability. This is nothing more than the law of large numbers. \end{example}$

Citation: admin. (2006, May 15). The Asymptotic Equipartition Property. Retrieved November 23, 2009, from Free Online Course Materials — USU OpenCourseWare Web site: http://ocw.usu.edu/Electrical_and_Computer_Engineering/Information_Theory/equi.htm.