The Asymptotic Equipartition Property

Document Actions

Introduction :: Convergence :: Data Compression :: Sets

Convergence in probability

Before jumping intro the theorem we need to discuss what it means to converge in probability. There are a variety of ways that things can converge in the world of probability. There is convergence in distribution (as exhibited by the central limit theorem), there is convergence almost surely (which is very strong), there is convergence in mean square, and there is convergence in probability. We could spend about three weeks working through what each of these mean, and which implies the other. However, for the moment we will simply focus on convergence in probability.

$\begin{definition}A sequence $X_1,X_2,\ldots,X_n$\ is said to {\bf converge in ... ...\rightarrow 0 \text{ as } n\rightarrow \infty. \end{displaymath}\end{definition}$
Recalling the definition of convergence of sequences, we can say this as follows: For any $\epsilon >0$ 0$" align="middle" border="0" height="28" width="39" /> and for any $\delta > 0$ 0$" align="middle" border="0" height="29" width="40" />, there is an n₀ such that

$\begin{displaymath}P[\vert X - X_n\vert < \epsilon] > 1-\delta \end{displaymath}$ < \epsilon] > 1-\delta \end{displaymath}" border="0" height="28" width="170" />

for all n > n₀.

$\begin{example} Let $X_i$\ be a sequence of iid random variables and let \begin... ...h}More briefly we might write $S_n \rightarrow S$\ in probability. \end{example}$
One way of quantifying how we are doing on the convergence is by means of Markov's inequality: For a positive r.v. and any $\delta > 0$ 0$" align="middle" border="0" height="29" width="40" />,

$\begin{displaymath}P[X \geq \epsilon] \leq \frac{EX}{\delta}. \end{displaymath}$

From this we can derive the Chebyshev inequality: for a r.v. Y with mean $\mu$ and variance $\sigma^2$

$\begin{displaymath}P[\vert Y - \mu\vert > \epsilon] \leq \frac{\sigma^2}{\epsilon^2}. \end{displaymath}$ \epsilon] \leq \frac{\sigma^2}{\epsilon^2}. \end{displaymath}" border="0" height="41" width="145" />

From this we can show convergence of the sample mean (the WLLN).

Now the AEP:
$\begin{theorem} (AEP) If $X_1,X_2,\ldots,X_n$\ are iid $\sim p(x)$\ then \begin... ...,X_2,\ldots,X_n) \rightarrow H(X) \end{displaymath}in probability. \end{theorem}$

$\begin{proof} Since the $X_i$\ are iid, so are the $\log p(X_i)$. By independen... ...(X)\text{ in probability} \\ &= H(X). \end{aligned}\end{displaymath}\end{proof}$

The set of sequences that come up most often (according the probability law of the r.v.) are typical sequences. The typicality is defined as follows:
$\begin{definition} The {\bf typical set} $A_\epsilon^{(n)}$\ with respect to $p(... ..._1,x_2,\ldots,x_n) \leq 2^{-n(H(X)-\epsilon)}. \end{displaymath}\end{definition}$
So the typical sequences occur with a probability that is in the neighborhood of 2^-nH.
$\begin{example} Returning to the first example, $H(X) = 0.88129$, and $nH(X) = ... ...o the probability with which the sequence with three ones occurs. \end{example}$
The typical set $A_\epsilon^{(n)}$ has the following properties:
$\begin{theorem} \begin{enumerate} \item If $(x_1,x_2,\ldots,x_n) \in A_\epsilo... ...2^{n(H(X)-\epsilon)}$ for $n$\ sufficiently large. \end{enumerate}\end{theorem}$
Interpretations: By (1), nearly all of the elements in the typical set are nearly equiprobable. By (2), the typical set occurs with probability near 1 (that is why it is typical!). By (3) and (4), the number of elements in the typical set is nearly 2^nH.

$\begin{proof} \begin{enumerate} \item Take $-\frac{1}{n} \log_2$\ of the defin... ... A_\epsilon^{(n)}\vert. \end{aligned}\end{displaymath}\end{enumerate}\end{proof}$

Copyright 2008, by the Contributing Authors. Cite/attribute Resource. admin. (2006, May 17). The Asymptotic Equipartition Property. Retrieved November 23, 2009, from Free Online Course Materials — USU OpenCourseWare Web site: http://ocw.usu.edu/Electrical_and_Computer_Engineering/Information_Theory/equi_1.htm. This work is licensed under a Creative Commons License.

Course Contents Information Theory Home About the Professor Syllabus Schedule Homework Assignments Programming Assignments Download This Course	Personal tools Log in You are here: Home → Electrical and Computer Engineering → Information Theory → The Asymptotic Equipartition Property
	Info The Asymptotic Equipartition Property Document Actions Introduction :: Convergence :: Data Compression :: Sets Convergence in probability Before jumping intro the theorem we need to discuss what it means to converge in probability. There are a variety of ways that things can converge in the world of probability. There is convergence in distribution (as exhibited by the central limit theorem), there is convergence almost surely (which is very strong), there is convergence in mean square, and there is convergence in probability. We could spend about three weeks working through what each of these mean, and which implies the other. However, for the moment we will simply focus on convergence in probability. $\begin{definition}A sequence $X_1,X_2,\ldots,X_n$\ is said to {\bf converge in ... ...\rightarrow 0 \text{ as } n\rightarrow \infty. \end{displaymath}\end{definition}$ Recalling the definition of convergence of sequences, we can say this as follows: For any $\epsilon >0$ 0$" align="middle" border="0" height="28" width="39" /> and for any $\delta > 0$ 0$" align="middle" border="0" height="29" width="40" />, there is an n₀ such that $\begin{displaymath}P[\vert X - X_n\vert < \epsilon] > 1-\delta \end{displaymath}$ < \epsilon] > 1-\delta \end{displaymath}" border="0" height="28" width="170" /> for all n > n₀. $\begin{example} Let $X_i$\ be a sequence of iid random variables and let \begin... ...h}More briefly we might write $S_n \rightarrow S$\ in probability. \end{example}$ One way of quantifying how we are doing on the convergence is by means of Markov's inequality: For a positive r.v. and any $\delta > 0$ 0$" align="middle" border="0" height="29" width="40" />, $\begin{displaymath}P[X \geq \epsilon] \leq \frac{EX}{\delta}. \end{displaymath}$ From this we can derive the Chebyshev inequality: for a r.v. Y with mean $\mu$ and variance $\sigma^2$ $\begin{displaymath}P[\vert Y - \mu\vert > \epsilon] \leq \frac{\sigma^2}{\epsilon^2}. \end{displaymath}$ \epsilon] \leq \frac{\sigma^2}{\epsilon^2}. \end{displaymath}" border="0" height="41" width="145" /> From this we can show convergence of the sample mean (the WLLN). Now the AEP: $\begin{theorem} (AEP) If $X_1,X_2,\ldots,X_n$\ are iid $\sim p(x)$\ then \begin... ...,X_2,\ldots,X_n) \rightarrow H(X) \end{displaymath}in probability. \end{theorem}$ $\begin{proof} Since the $X_i$\ are iid, so are the $\log p(X_i)$. By independen... ...(X)\text{ in probability} \\ &= H(X). \end{aligned}\end{displaymath}\end{proof}$ The set of sequences that come up most often (according the probability law of the r.v.) are typical sequences. The typicality is defined as follows: $\begin{definition} The {\bf typical set} $A_\epsilon^{(n)}$\ with respect to $p(... ..._1,x_2,\ldots,x_n) \leq 2^{-n(H(X)-\epsilon)}. \end{displaymath}\end{definition}$ So the typical sequences occur with a probability that is in the neighborhood of 2^-nH. $\begin{example} Returning to the first example, $H(X) = 0.88129$, and $nH(X) = ... ...o the probability with which the sequence with three ones occurs. \end{example}$ The typical set $A_\epsilon^{(n)}$ has the following properties: $\begin{theorem} \begin{enumerate} \item If $(x_1,x_2,\ldots,x_n) \in A_\epsilo... ...2^{n(H(X)-\epsilon)}$ for $n$\ sufficiently large. \end{enumerate}\end{theorem}$ Interpretations: By (1), nearly all of the elements in the typical set are nearly equiprobable. By (2), the typical set occurs with probability near 1 (that is why it is typical!). By (3) and (4), the number of elements in the typical set is nearly 2^nH. $\begin{proof} \begin{enumerate} \item Take $-\frac{1}{n} \log_2$\ of the defin... ... A_\epsilon^{(n)}\vert. \end{aligned}\end{displaymath}\end{enumerate}\end{proof}$ Copyright 2008, by the Contributing Authors. Cite/attribute Resource. admin. (2006, May 17). The Asymptotic Equipartition Property. Retrieved November 23, 2009, from Free Online Course Materials — USU OpenCourseWare Web site: http://ocw.usu.edu/Electrical_and_Computer_Engineering/Information_Theory/equi_1.htm. This work is licensed under a Creative Commons License.	Reuse Course Download this course

Sections

Personal tools

The Asymptotic Equipartition Property

Document Actions

Convergence in probability