Personal tools
  •  

Channel Capacity

Document Actions
  • Content View
  • Bookmarks
  • CourseFeed

Definitions   ::   Symmetric Channels   ::   Closer Look   ::   Typical Sequences   ::   Theorem

Jointly typical sequences

Recall the definition of a typical sequence: it is the sequence we expect (probabilistically) to occur, given the statistics of the source. We had a theorem regarding the approximate number of typical sequences, the probability of a typical sequence, and so forth. In this section we generalize this.

All along this semester we have used the notion of representing sequences of random variables as vectors. For example, the r.v.'s ( X 1 , X 2 ) we could represent with a vector-valued random variable X . In a sense, this is all that we are doing with the jointly-typical sequences. We consider sequences ( x n , y n ) as if there were simply some sequence z n , and ask: for the set of sequences z n , what are the typical sequences:

\begin{definition}
The set $A_\epsilon^{(n)}$\ of {\bf jointly typical } sequen...
...Y)\right\vert <
\epsilon\right\}
\end{aligned}\end{displaymath}\end{definition}

From this definition, we can conclude similar sorts of things about jointly typical sequences as we can about typical sequences:

\begin{theorem}
(Joint AEP).
\begin{enumerate}
\item $\text{Pr}((X^n,Y^n) \in...
...epsilon) 2^{-n(I(X;Y)+3\epsilon)}.
\end{displaymath}\end{enumerate}\end{theorem}

\begin{proof}
By WLLN,
\begin{displaymath}-\frac{1}{n}\log p(X^n) \rightarrow -...
...e
sum). Similar arguments can be used to derive the last inequality.
\end{proof}

There are about 2 nH ( X ) typical X sequences, about 2 nH ( Y ) typical Y sequences, and only about 2 nH ( X , Y ) jointly typical sequences. This means that if we choose a typical X sequence and independently choose a typical Y sequence (without regard to the X sequence), in not all cases will the sequence ( X n , Y n ) so chosen be jointly typical. In fact, from the last part of the theorem, the probability that the sequences chosen independently will be jointly typical is about 2 - nI ( X ; Y ) . This means that we would have to try (at random) about 2 nI ( X ; Y ) sequence pairs before we choose a jointly typical pair. Thinking now in terms of fixing Y and choosing X at random, this suggests that there are about 2 nI ( X ; Y ) distinguishable sequences in X .

Copyright 2008, by the Contributing Authors. Cite/attribute Resource . admin. (2006, May 17). Channel Capacity. Retrieved January 07, 2011, from Free Online Course Materials — USU OpenCourseWare Web site: http://ocw.usu.edu/Electrical_and_Computer_Engineering/Information_Theory/lecture8_4.htm. This work is licensed under a Creative Commons License Creative Commons License