The Asymptotic Equipartition Property

Document Actions

Introduction :: Convergence :: Data Compression :: Sets

Where does this lead: data compression

Typical sequences occur most of the time. Therefore, one way to perform data compression is to only provide efficient codes for the typical sequences, and other (less efficient) codes for the non-typical sequences. Since there are $\leq 2^{n(H+\epsilon)}$ sequences in $A_\epsilon^{(n)}$ , then we can encode every one of them with no more than $n(H+\epsilon)+1$ bits. Let us prefix each typical-sequence code with a 0, so the total length of codes for typical sequences is $\leq n(H+\epsilon) + 2$ .

Each sequence not in $A_\epsilon^{(n)}$ can be indexed with not more than $n \log\vert\Xc\vert+1$ bits. If these codes are prefixed with a 1, then we have a code for all sequences in $\Xc^n$ . Even though we are doing a brute-force enumeration of the atypical set (overlooking the fact that we don't need to do those codes that are already in the typical set), we can still do a good job (on the average).

Notation: let xⁿ denote the sequence $x_1,x_2,\ldots,x_n$ .
$\begin{theorem} Let $X^n$\ be i.i.d., and let $\epsilon > 0$. Then there exists... ...\leq H(X) + \epsilon \end{displaymath}for $n$\ sufficiently large. \end{theorem}$ 0$. Then there exists... ...\leq H(X) + \epsilon \end{displaymath}for $n$\ sufficiently large. \end{theorem}" align="bottom" border="0" height="104" width="556" />
This theorem does not go so far yet as to say that this is about the best that we can do.

$\begin{proof} The expected length of the codeword is \begin{displaymath}\begin{... ...l as desired by choice of $\epsilon$\ followed by choice of $n$. \par\end{proof}$

Copyright 2008, by the Contributing Authors. Cite/attribute Resource. admin. (2006, May 17). The Asymptotic Equipartition Property. Retrieved November 23, 2009, from Free Online Course Materials — USU OpenCourseWare Web site: http://ocw.usu.edu/Electrical_and_Computer_Engineering/Information_Theory/equi_2.htm. This work is licensed under a Creative Commons License.

Course Contents Information Theory Home About the Professor Syllabus Schedule Homework Assignments Programming Assignments Download This Course	Personal tools Log in You are here: Home → Electrical and Computer Engineering → Information Theory → The Asymptotic Equipartition Property
	Info The Asymptotic Equipartition Property Document Actions Introduction :: Convergence :: Data Compression :: Sets Where does this lead: data compression Typical sequences occur most of the time. Therefore, one way to perform data compression is to only provide efficient codes for the typical sequences, and other (less efficient) codes for the non-typical sequences. Since there are $\leq 2^{n(H+\epsilon)}$ sequences in $A_\epsilon^{(n)}$ , then we can encode every one of them with no more than $n(H+\epsilon)+1$ bits. Let us prefix each typical-sequence code with a 0, so the total length of codes for typical sequences is $\leq n(H+\epsilon) + 2$ . Each sequence not in $A_\epsilon^{(n)}$ can be indexed with not more than $n \log\vert\Xc\vert+1$ bits. If these codes are prefixed with a 1, then we have a code for all sequences in $\Xc^n$ . Even though we are doing a brute-force enumeration of the atypical set (overlooking the fact that we don't need to do those codes that are already in the typical set), we can still do a good job (on the average). Notation: let xⁿ denote the sequence $x_1,x_2,\ldots,x_n$ . $\begin{theorem} Let $X^n$\ be i.i.d., and let $\epsilon > 0$. Then there exists... ...\leq H(X) + \epsilon \end{displaymath}for $n$\ sufficiently large. \end{theorem}$ 0$. Then there exists... ...\leq H(X) + \epsilon \end{displaymath}for $n$\ sufficiently large. \end{theorem}" align="bottom" border="0" height="104" width="556" /> This theorem does not go so far yet as to say that this is about the best that we can do. $\begin{proof} The expected length of the codeword is \begin{displaymath}\begin{... ...l as desired by choice of $\epsilon$\ followed by choice of $n$. \par\end{proof}$ Copyright 2008, by the Contributing Authors. Cite/attribute Resource. admin. (2006, May 17). The Asymptotic Equipartition Property. Retrieved November 23, 2009, from Free Online Course Materials — USU OpenCourseWare Web site: http://ocw.usu.edu/Electrical_and_Computer_Engineering/Information_Theory/equi_2.htm. This work is licensed under a Creative Commons License.	Reuse Course Download this course

Sections

Personal tools

The Asymptotic Equipartition Property

Document Actions

Where does this lead: data compression