Personal tools
You are here: Home → Electrical and Computer Engineering → Information Theory → The Asymptotic Equipartition Property

The Asymptotic Equipartition Property

Document Actions
  • Send this
  • Print this
  • Content View
  • Bookmarks
  • CourseFeed

Introduction   ::   Convergence   ::   Data Compression   ::   Sets

Introduction

The AEP is the weak law of large numbers as applied to the estimation of entropy.

The law of large numbers and AEP

The law of large numbers states that for i.i.d. random variables $X_1,X_2,\ldots,X_n$ the sum

\begin{displaymath}\frac{1}{n}\sum_{i=1}^n X_i \end{displaymath}

is close to its expected value EX. (How close? How can we tell?) The AEP states that for i.i.d. r.v.'s,

\begin{displaymath}\frac{1}{n}\log \frac{1}{p(X_1,X_2,\ldots,X_n)} \end{displaymath}

is close to H(X). To put it another way,

\begin{displaymath}p(x_1,x_2,\ldots,x_n) \approx 2^{-nH} \end{displaymath}


\begin{example} Suppose $X \in \{ \zerobf,\onebf\}$, with $p(\onebf) = p$, $p(... ...e probability. This is nothing more than the law of large numbers. \end{example}


Copyright 2008, Todd Moon. Cite/attribute Resource. admin. (2006, May 15). The Asymptotic Equipartition Property. Retrieved November 08, 2009, from Free Online Course Materials — USU OpenCourseWare Web site: http://ocw.usu.edu/Electrical_and_Computer_Engineering/Information_Theory/equi.htm. This work is licensed under a Creative Commons License. Creative Commons License
Reuse Course
Download this course