Personal tools
  •  
You are here: Home Electrical and Computer Engineering Information Theory The Asymptotic Equipartition Property

The Asymptotic Equipartition Property

Document Actions
  • Content View
  • Bookmarks
  • CourseFeed

Introduction   ::   Convergence   ::   Data Compression   ::   Sets

Where does this lead: data compression

Typical sequences occur most of the time. Therefore, one way to perform data compression is to only provide efficient codes for the typical sequences, and other (less efficient) codes for the non-typical sequences. Since there are $\leq 2^{n(H+\epsilon)}$ sequences in $A_\epsilon^{(n)}$ , then we can encode every one of them with no more than $n(H+\epsilon)+1$ bits. Let us prefix each typical-sequence code with a 0, so the total length of codes for typical sequences is $\leq
n(H+\epsilon) + 2$ .

Each sequence not in $A_\epsilon^{(n)}$ can be indexed with not more than $n \log\vert\Xc\vert+1$ bits. If these codes are prefixed with a 1, then we have a code for all sequences in $\Xc^n$ . Even though we are doing a brute-force enumeration of the atypical set (overlooking the fact that we don't need to do those codes that are already in the typical set), we can still do a good job (on the average).

Notation: let x n denote the sequence $x_1,x_2,\ldots,x_n$ .
\begin{theorem}
Let $X^n$\ be i.i.d., and let $\epsilon > 0$. Then there exists...
...\leq H(X) + \epsilon
\end{displaymath}for $n$\ sufficiently large.
\end{theorem}
This theorem does not go so far yet as to say that this is about the best that we can do.


\begin{proof}
The expected length of the codeword is
\begin{displaymath}\begin{...
...l as desired by choice of $\epsilon$\ followed by
choice of $n$.
\par\end{proof}

Copyright 2008, by the Contributing Authors. Cite/attribute Resource . admin. (2006, May 17). The Asymptotic Equipartition Property. Retrieved January 07, 2011, from Free Online Course Materials — USU OpenCourseWare Web site: http://ocw.usu.edu/Electrical_and_Computer_Engineering/Information_Theory/equi_2.htm. This work is licensed under a Creative Commons License Creative Commons License