Personal tools
You are here: Home Electrical and Computer Engineering Information Theory Application of Information Theory to Blind Source Separation

Application of Information Theory to Blind Source Separation

Document Actions
  • Content View
  • Bookmarks
  • CourseFeed

Introduction   ::   BSS   ::   Mackay's Approach   ::   Natural Gradient   ::   p(u)

Mackay's approach

We consider the BSS problem as a ML estimation problem. To begin with, we state a lemma that we will need.

Let $\xbf = A\sbf$, so that $x_j = \sum_{i} v_{ji} s_i$. Assume
...{\det(A)} \prod_j p_j(\sum_i(A^{-1})_{ji}

We shall give an explicit proof for the $\matsize{2}{2}$\ case.
from which the result is apparent for this case.
Given a sequence of observed data $\{\xbf_1, \ldots, \xbf_N\}$ obtained from $\xbf = A \sbf$ we write down the joint probability as

\begin{displaymath}P(\xbf_1, \ldots, \xbf_N,\sbf_1,\ldots, \sbf_N\vert A)
= \pr...
... - \sum_i a_{ji} \sbf_{n,i}) \prod_j

To do ML estimation of A we examine

\begin{displaymath}P(\xbf_1,\ldots, \xbf_N\vert A) = \prod_n P(\xbf_n\vert A),

The ML solution is to find the A that maximizes this likelihood function. We have

P(\xbf_n\vert A) &= \int P(\xbf_n\vert\sbf_...
...)} \prod_j p_j(\sum_i(A^{-1})_{ji}\xbf_{n,i}),

where the lemma we derived before has been used. The log likelihood function is

\begin{displaymath}\log P(\xbf_n\vert A) = -\log \det A + \sum_j \log

Let W = A -1 . Then

\begin{displaymath}\log P(\xbf_n\vert A) = \log \det W + \sum_j \log
p_j(\sum_i W_{ji}\xbf_{n,i}).

Now we proceed as before, computing the derivative of the log likelihood with respect to W . The first part is easy:

\begin{displaymath}\partiald{}{W} \log \det W = W^{-T}.

For the first part:

\partiald{}{W_{mn}} \sum_j \log p_j(\sum_i W...
...frac{1}{p_m(a_m)} x_n \partiald{p_m(a_m)}{a_m}


\begin{displaymath}\phi_m(a_m) = \frac{d}{d a_m} \log p_m(a_m) = \frac{1}{p_m(a_m)}
\frac{d p_m(a_m)}{a_m}

and let $z_m = \phi_m(a_m)$ . Then we have

\begin{displaymath}\partiald{}{W_{mn}}\sum_j \log p_j(\sum_i W_{ji} x_i) = x_n z_m.


\begin{displaymath}\partiald{}{W} \log P(\xbf_n\vert A) = W^{-T} + \zbf \xbf^T.

Compare with what we had before!

Copyright 2008, by the Contributing Authors. Cite/attribute Resource . admin. (2006, May 17). Application of Information Theory to Blind Source Separation. Retrieved January 07, 2011, from Free Online Course Materials — USU OpenCourseWare Web site: This work is licensed under a Creative Commons License Creative Commons License