Personal tools
  •  
You are here: Home Electrical and Computer Engineering Information Theory Application of Information Theory to Blind Source Separation

Application of Information Theory to Blind Source Separation

Document Actions
  • Content View
  • Bookmarks
  • CourseFeed

Introduction   ::   BSS   ::   Mackay's Approach   ::   Natural Gradient   ::   p(u)

Mackay's approach

We consider the BSS problem as a ML estimation problem. To begin with, we state a lemma that we will need.


\begin{lemma}
Let $\xbf = A\sbf$, so that $x_j = \sum_{i} v_{ji} s_i$. Assume
...
...{\det(A)} \prod_j p_j(\sum_i(A^{-1})_{ji}
x_i)
\end{displaymath}
\end{lemma}

\begin{proof}
We shall give an explicit proof for the $\matsize{2}{2}$\ case.
...
...\end{displaymath}
from which the result is apparent for this case.
\end{proof}
Given a sequence of observed data $\{\xbf_1, \ldots, \xbf_N\}$ obtained from $\xbf = A \sbf$ we write down the joint probability as

\begin{displaymath}P(\xbf_1, \ldots, \xbf_N,\sbf_1,\ldots, \sbf_N\vert A)
= \pr...
... - \sum_i a_{ji} \sbf_{n,i}) \prod_j
p_j(\sbf_{n,j})\right].
\end{displaymath}

To do ML estimation of A we examine

\begin{displaymath}P(\xbf_1,\ldots, \xbf_N\vert A) = \prod_n P(\xbf_n\vert A),
\end{displaymath}

The ML solution is to find the A that maximizes this likelihood function. We have

\begin{displaymath}\begin{aligned}
P(\xbf_n\vert A) &= \int P(\xbf_n\vert\sbf_...
...)} \prod_j p_j(\sum_i(A^{-1})_{ji}\xbf_{n,i}),
\end{aligned}
\end{displaymath}

where the lemma we derived before has been used. The log likelihood function is

\begin{displaymath}\log P(\xbf_n\vert A) = -\log \det A + \sum_j \log
p_j(\sum_i(A^{-1})_{ji}\xbf_{n,i}).
\end{displaymath}

Let W = A -1 . Then

\begin{displaymath}\log P(\xbf_n\vert A) = \log \det W + \sum_j \log
p_j(\sum_i W_{ji}\xbf_{n,i}).
\end{displaymath}

Now we proceed as before, computing the derivative of the log likelihood with respect to W . The first part is easy:

\begin{displaymath}\partiald{}{W} \log \det W = W^{-T}.
\end{displaymath}

For the first part:

\begin{displaymath}\begin{aligned}
\partiald{}{W_{mn}} \sum_j \log p_j(\sum_i W...
...frac{1}{p_m(a_m)} x_n \partiald{p_m(a_m)}{a_m}
\end{aligned}
\end{displaymath}

Let

\begin{displaymath}\phi_m(a_m) = \frac{d}{d a_m} \log p_m(a_m) = \frac{1}{p_m(a_m)}
\frac{d p_m(a_m)}{a_m}
\end{displaymath}

and let $z_m = \phi_m(a_m)$ . Then we have

\begin{displaymath}\partiald{}{W_{mn}}\sum_j \log p_j(\sum_i W_{ji} x_i) = x_n z_m.
\end{displaymath}

Thus

\begin{displaymath}\partiald{}{W} \log P(\xbf_n\vert A) = W^{-T} + \zbf \xbf^T.
\end{displaymath}

Compare with what we had before!

Copyright 2008, by the Contributing Authors. Cite/attribute Resource . admin. (2006, May 17). Application of Information Theory to Blind Source Separation. Retrieved January 07, 2011, from Free Online Course Materials — USU OpenCourseWare Web site: http://ocw.usu.edu/Electrical_and_Computer_Engineering/Information_Theory/lecture4_3.htm. This work is licensed under a Creative Commons License Creative Commons License