Channel Capacity

Document Actions

Definitions :: Symmetric Channels :: Closer Look :: Typical Sequences :: Theorem

Jointly typical sequences

Recall the definition of a typical sequence: it is the sequence we expect (probabilistically) to occur, given the statistics of the source. We had a theorem regarding the approximate number of typical sequences, the probability of a typical sequence, and so forth. In this section we generalize this.

All along this semester we have used the notion of representing sequences of random variables as vectors. For example, the r.v.'s (X₁,X₂) we could represent with a vector-valued random variable X. In a sense, this is all that we are doing with the jointly-typical sequences. We consider sequences (xⁿ,yⁿ) as if there were simply some sequence zⁿ, and ask: for the set of sequences zⁿ, what are the typical sequences:

$\begin{definition} The set $A_\epsilon^{(n)}$\ of {\bf jointly typical } sequen... ...Y)\right\vert < \epsilon\right\} \end{aligned}\end{displaymath}\end{definition}$

From this definition, we can conclude similar sorts of things about jointly typical sequences as we can about typical sequences:

$\begin{theorem} (Joint AEP). \begin{enumerate} \item $\text{Pr}((X^n,Y^n) \in... ...epsilon) 2^{-n(I(X;Y)+3\epsilon)}. \end{displaymath}\end{enumerate}\end{theorem}$

$\begin{proof} By WLLN, \begin{displaymath}-\frac{1}{n}\log p(X^n) \rightarrow -... ...e sum). Similar arguments can be used to derive the last inequality. \end{proof}$

There are about 2^nH(X) typical X sequences, about 2^nH(Y) typical Y sequences, and only about 2^nH(X,Y) jointly typical sequences. This means that if we choose a typical X sequence and independently choose a typical Y sequence (without regard to the X sequence), in not all cases will the sequence (Xⁿ,Yⁿ) so chosen be jointly typical. In fact, from the last part of the theorem, the probability that the sequences chosen independently will be jointly typical is about 2^-nI(X;Y). This means that we would have to try (at random) about 2^nI(X;Y) sequence pairs before we choose a jointly typical pair. Thinking now in terms of fixing Y and choosing X at random, this suggests that there are about 2^nI(X;Y) distinguishable sequences in X.

Copyright 2008, by the Contributing Authors. Cite/attribute Resource. admin. (2006, May 17). Channel Capacity. Retrieved November 22, 2009, from Free Online Course Materials — USU OpenCourseWare Web site: http://ocw.usu.edu/Electrical_and_Computer_Engineering/Information_Theory/lecture8_4.htm. This work is licensed under a Creative Commons License.

Course Contents Information Theory Home About the Professor Syllabus Schedule Homework Assignments Programming Assignments Download This Course	Personal tools Log in You are here: Home → Electrical and Computer Engineering → Information Theory → Channel Capacity
	Info Channel Capacity Document Actions Definitions :: Symmetric Channels :: Closer Look :: Typical Sequences :: Theorem Jointly typical sequences Recall the definition of a typical sequence: it is the sequence we expect (probabilistically) to occur, given the statistics of the source. We had a theorem regarding the approximate number of typical sequences, the probability of a typical sequence, and so forth. In this section we generalize this. All along this semester we have used the notion of representing sequences of random variables as vectors. For example, the r.v.'s (X₁,X₂) we could represent with a vector-valued random variable X. In a sense, this is all that we are doing with the jointly-typical sequences. We consider sequences (xⁿ,yⁿ) as if there were simply some sequence zⁿ, and ask: for the set of sequences zⁿ, what are the typical sequences: $\begin{definition} The set $A_\epsilon^{(n)}$\ of {\bf jointly typical } sequen... ...Y)\right\vert < \epsilon\right\} \end{aligned}\end{displaymath}\end{definition}$ From this definition, we can conclude similar sorts of things about jointly typical sequences as we can about typical sequences: $\begin{theorem} (Joint AEP). \begin{enumerate} \item $\text{Pr}((X^n,Y^n) \in... ...epsilon) 2^{-n(I(X;Y)+3\epsilon)}. \end{displaymath}\end{enumerate}\end{theorem}$ $\begin{proof} By WLLN, \begin{displaymath}-\frac{1}{n}\log p(X^n) \rightarrow -... ...e sum). Similar arguments can be used to derive the last inequality. \end{proof}$ There are about 2^nH(X) typical X sequences, about 2^nH(Y) typical Y sequences, and only about 2^nH(X,Y) jointly typical sequences. This means that if we choose a typical X sequence and independently choose a typical Y sequence (without regard to the X sequence), in not all cases will the sequence (Xⁿ,Yⁿ) so chosen be jointly typical. In fact, from the last part of the theorem, the probability that the sequences chosen independently will be jointly typical is about 2^-nI(X;Y). This means that we would have to try (at random) about 2^nI(X;Y) sequence pairs before we choose a jointly typical pair. Thinking now in terms of fixing Y and choosing X at random, this suggests that there are about 2^nI(X;Y) distinguishable sequences in X. Copyright 2008, by the Contributing Authors. Cite/attribute Resource. admin. (2006, May 17). Channel Capacity. Retrieved November 22, 2009, from Free Online Course Materials — USU OpenCourseWare Web site: http://ocw.usu.edu/Electrical_and_Computer_Engineering/Information_Theory/lecture8_4.htm. This work is licensed under a Creative Commons License.	Reuse Course Download this course

Sections

Personal tools

Channel Capacity

Document Actions

Jointly typical sequences