Personal tools
  •  
You are here: Home Electrical and Computer Engineering Information Theory The Asymptotic Equipartition Property

The Asymptotic Equipartition Property

Document Actions
  • Content View
  • Bookmarks
  • CourseFeed

Introduction   ::   Convergence   ::   Data Compression   ::   Sets

High probability sets and the typical set

We introduce here some notation that is useful in later chapters.
\begin{example}
We begin with an example. Suppose $a_n$\ is a sequence defined a...
... of e, to be precise), the log of the ratio doesn't grow too
fast.
\end{example}

\begin{example}
We present another example of the same concept. Suppose $a_n =
...
... between $a_n$\ and $b_n$\ is increasing, it
is increasing slowly.
\end{example}

\begin{example}
Now let $a_n = e^{4n}$\ and $b_n = e^{3n}$. Then
\begin{display...
...c{a_n}{b_n} = 1.
\end{displaymath}So the ratio increases too fast.
\end{example}
By these examples, we are in a better position to understand the following definition.
\begin{definition}
The notation $a_n \dot{=} b_n$\ means
\begin{displaymath}\li...
..., $a_n$\ and $b_n$\ {\bf agree to first order in the exponent}.
\end{definition}

Now, the typical set $A_\epsilon^{(n)}$ is a fairly small set that contains most of the probability, but it is not clear if it is the smallest such set. We will show that any small but highly probable set must contain a significant overlap with the typical set.
\begin{definition}
Let $B_\delta^{(n)} \in \Xc^n$\ be any set with
\begin{disp...
...}$B_\delta^{(n)}$\ may be viewed as a \lq\lq high probability set.''
\end{definition}
Then we have the following theorem:
\begin{theorem}
Let $X_i$\ be i.i.d. and $\sim p(x)$. For $\delta < 1/2$\ and a...
...{=} \vert A_{\epsilon}^{(n)}\vert \dot{=} 2^{nH}.
\end{displaymath}\end{theorem}
To contrast, consider X i be Bernoulli(.9). Then a typical set contains sequences in which the proportion of 1's is close to .9. But it does not contain the single most probable sequence, the sequence of all 1s. The set $B_\delta^{(n)}$ (the high probability set) will include all the most probable sequences, including the sequence of all 1s. By the theorem, A and B must both contain the sequences that are about 90% 1s, and that the two sets are almost equal in size.

Copyright 2008, by the Contributing Authors. Cite/attribute Resource . admin. (2006, May 17). The Asymptotic Equipartition Property. Retrieved January 07, 2011, from Free Online Course Materials — USU OpenCourseWare Web site: http://ocw.usu.edu/Electrical_and_Computer_Engineering/Information_Theory/equi_3.htm. This work is licensed under a Creative Commons License Creative Commons License