The Asymptotic Equipartition Property

Document Actions

Introduction :: Convergence :: Data Compression :: Sets

High probability sets and the typical set

We introduce here some notation that is useful in later chapters.
$\begin{example} We begin with an example. Suppose $a_n$\ is a sequence defined a... ... of e, to be precise), the log of the ratio doesn't grow too fast. \end{example}$

$\begin{example} We present another example of the same concept. Suppose $a_n = ... ... between $a_n$\ and $b_n$\ is increasing, it is increasing slowly. \end{example}$

$\begin{example} Now let $a_n = e^{4n}$\ and $b_n = e^{3n}$. Then \begin{display... ...c{a_n}{b_n} = 1. \end{displaymath}So the ratio increases too fast. \end{example}$
By these examples, we are in a better position to understand the following definition.
$\begin{definition} The notation $a_n \dot{=} b_n$\ means \begin{displaymath}\li... ..., $a_n$\ and $b_n$\ {\bf agree to first order in the exponent}. \end{definition}$

Now, the typical set $A_\epsilon^{(n)}$ is a fairly small set that contains most of the probability, but it is not clear if it is the smallest such set. We will show that any small but highly probable set must contain a significant overlap with the typical set.
$\begin{definition} Let $B_\delta^{(n)} \in \Xc^n$\ be any set with \begin{disp... ...}$B_\delta^{(n)}$\ may be viewed as a \lq\lq high probability set.'' \end{definition}$
Then we have the following theorem:
$\begin{theorem} Let $X_i$\ be i.i.d. and $\sim p(x)$. For $\delta < 1/2$\ and a... ...{=} \vert A_{\epsilon}^{(n)}\vert \dot{=} 2^{nH}. \end{displaymath}\end{theorem}$ < 1/2$\ and a... ...{=} \vert A_{\epsilon}^{(n)}\vert \dot{=} 2^{nH}. \end{displaymath}\end{theorem}" align="bottom" border="0" height="161" width="558" />
To contrast, consider X_i be Bernoulli(.9). Then a typical set contains sequences in which the proportion of 1's is close to .9. But it does not contain the single most probable sequence, the sequence of all 1s. The set $B_\delta^{(n)}$ (the high probability set) will include all the most probable sequences, including the sequence of all 1s. By the theorem, A and B must both contain the sequences that are about 90% 1s, and that the two sets are almost equal in size.

Copyright 2008, by the Contributing Authors. Cite/attribute Resource. admin. (2006, May 17). The Asymptotic Equipartition Property. Retrieved November 23, 2009, from Free Online Course Materials — USU OpenCourseWare Web site: http://ocw.usu.edu/Electrical_and_Computer_Engineering/Information_Theory/equi_3.htm. This work is licensed under a Creative Commons License.

Course Contents Information Theory Home About the Professor Syllabus Schedule Homework Assignments Programming Assignments Download This Course	Personal tools Log in You are here: Home → Electrical and Computer Engineering → Information Theory → The Asymptotic Equipartition Property
	Info The Asymptotic Equipartition Property Document Actions Introduction :: Convergence :: Data Compression :: Sets High probability sets and the typical set We introduce here some notation that is useful in later chapters. $\begin{example} We begin with an example. Suppose $a_n$\ is a sequence defined a... ... of e, to be precise), the log of the ratio doesn't grow too fast. \end{example}$ $\begin{example} We present another example of the same concept. Suppose $a_n = ... ... between $a_n$\ and $b_n$\ is increasing, it is increasing slowly. \end{example}$ $\begin{example} Now let $a_n = e^{4n}$\ and $b_n = e^{3n}$. Then \begin{display... ...c{a_n}{b_n} = 1. \end{displaymath}So the ratio increases too fast. \end{example}$ By these examples, we are in a better position to understand the following definition. $\begin{definition} The notation $a_n \dot{=} b_n$\ means \begin{displaymath}\li... ..., $a_n$\ and $b_n$\ {\bf agree to first order in the exponent}. \end{definition}$ Now, the typical set $A_\epsilon^{(n)}$ is a fairly small set that contains most of the probability, but it is not clear if it is the smallest such set. We will show that any small but highly probable set must contain a significant overlap with the typical set. $\begin{definition} Let $B_\delta^{(n)} \in \Xc^n$\ be any set with \begin{disp... ...}$B_\delta^{(n)}$\ may be viewed as a \lq\lq high probability set.'' \end{definition}$ Then we have the following theorem: $\begin{theorem} Let $X_i$\ be i.i.d. and $\sim p(x)$. For $\delta < 1/2$\ and a... ...{=} \vert A_{\epsilon}^{(n)}\vert \dot{=} 2^{nH}. \end{displaymath}\end{theorem}$ < 1/2$\ and a... ...{=} \vert A_{\epsilon}^{(n)}\vert \dot{=} 2^{nH}. \end{displaymath}\end{theorem}" align="bottom" border="0" height="161" width="558" /> To contrast, consider X_i be Bernoulli(.9). Then a typical set contains sequences in which the proportion of 1's is close to .9. But it does not contain the single most probable sequence, the sequence of all 1s. The set $B_\delta^{(n)}$ (the high probability set) will include all the most probable sequences, including the sequence of all 1s. By the theorem, A and B must both contain the sequences that are about 90% 1s, and that the two sets are almost equal in size. Copyright 2008, by the Contributing Authors. Cite/attribute Resource. admin. (2006, May 17). The Asymptotic Equipartition Property. Retrieved November 23, 2009, from Free Online Course Materials — USU OpenCourseWare Web site: http://ocw.usu.edu/Electrical_and_Computer_Engineering/Information_Theory/equi_3.htm. This work is licensed under a Creative Commons License.	Reuse Course Download this course

Sections

Personal tools

The Asymptotic Equipartition Property

Document Actions

High probability sets and the typical set