Application of Information Theory to Blind Source Separation

Document Actions

Introduction :: BSS :: Mackay's Approach :: Natural Gradient :: p(u)

Mackay's approach

We consider the BSS problem as a ML estimation problem. To begin with, we state a lemma that we will need.

$\begin{lemma} Let $\xbf = A\sbf$, so that $x_j = \sum_{i} v_{ji} s_i$. Assume ... ...{\det(A)} \prod_j p_j(\sum_i(A^{-1})_{ji} x_i) \end{displaymath} \end{lemma}$

$\begin{proof} We shall give an explicit proof for the $\matsize{2}{2}$\ case. ... ...\end{displaymath} from which the result is apparent for this case. \end{proof}$
Given a sequence of observed data $\{\xbf_1, \ldots, \xbf_N\}$ obtained from $\xbf = A \sbf$ we write down the joint probability as

$\begin{displaymath}P(\xbf_1, \ldots, \xbf_N,\sbf_1,\ldots, \sbf_N\vert A) = \pr... ... - \sum_i a_{ji} \sbf_{n,i}) \prod_j p_j(\sbf_{n,j})\right]. \end{displaymath}$

To do ML estimation of A we examine

$\begin{displaymath}P(\xbf_1,\ldots, \xbf_N\vert A) = \prod_n P(\xbf_n\vert A), \end{displaymath}$

The ML solution is to find the A that maximizes this likelihood function. We have

$\begin{displaymath}\begin{aligned} P(\xbf_n\vert A) &= \int P(\xbf_n\vert\sbf_... ...)} \prod_j p_j(\sum_i(A^{-1})_{ji}\xbf_{n,i}), \end{aligned} \end{displaymath}$

where the lemma we derived before has been used. The log likelihood function is

$\begin{displaymath}\log P(\xbf_n\vert A) = -\log \det A + \sum_j \log p_j(\sum_i(A^{-1})_{ji}\xbf_{n,i}). \end{displaymath}$

Let W = A^-1. Then

$\begin{displaymath}\log P(\xbf_n\vert A) = \log \det W + \sum_j \log p_j(\sum_i W_{ji}\xbf_{n,i}). \end{displaymath}$

Now we proceed as before, computing the derivative of the log likelihood with respect to W. The first part is easy:

$\begin{displaymath}\partiald{}{W} \log \det W = W^{-T}. \end{displaymath}$

For the first part:

$\begin{displaymath}\begin{aligned} \partiald{}{W_{mn}} \sum_j \log p_j(\sum_i W... ...frac{1}{p_m(a_m)} x_n \partiald{p_m(a_m)}{a_m} \end{aligned} \end{displaymath}$

Let

$\begin{displaymath}\phi_m(a_m) = \frac{d}{d a_m} \log p_m(a_m) = \frac{1}{p_m(a_m)} \frac{d p_m(a_m)}{a_m} \end{displaymath}$

and let $z_m = \phi_m(a_m)$ . Then we have

$\begin{displaymath}\partiald{}{W_{mn}}\sum_j \log p_j(\sum_i W_{ji} x_i) = x_n z_m. \end{displaymath}$

Thus

$\begin{displaymath}\partiald{}{W} \log P(\xbf_n\vert A) = W^{-T} + \zbf \xbf^T. \end{displaymath}$

Compare with what we had before!

Copyright 2008, by the Contributing Authors. Cite/attribute Resource. admin. (2006, May 17). Application of Information Theory to Blind Source Separation. Retrieved November 23, 2009, from Free Online Course Materials — USU OpenCourseWare Web site: http://ocw.usu.edu/Electrical_and_Computer_Engineering/Information_Theory/lecture4_3.htm. This work is licensed under a Creative Commons License.

Course Contents Information Theory Home About the Professor Syllabus Schedule Homework Assignments Programming Assignments Download This Course	Personal tools Log in You are here: Home → Electrical and Computer Engineering → Information Theory → Application of Information Theory to Blind Source Separation
	Info Application of Information Theory to Blind Source Separation Document Actions Introduction :: BSS :: Mackay's Approach :: Natural Gradient :: p(u) Mackay's approach We consider the BSS problem as a ML estimation problem. To begin with, we state a lemma that we will need. $\begin{lemma} Let $\xbf = A\sbf$, so that $x_j = \sum_{i} v_{ji} s_i$. Assume ... ...{\det(A)} \prod_j p_j(\sum_i(A^{-1})_{ji} x_i) \end{displaymath} \end{lemma}$ $\begin{proof} We shall give an explicit proof for the $\matsize{2}{2}$\ case. ... ...\end{displaymath} from which the result is apparent for this case. \end{proof}$ Given a sequence of observed data $\{\xbf_1, \ldots, \xbf_N\}$ obtained from $\xbf = A \sbf$ we write down the joint probability as $\begin{displaymath}P(\xbf_1, \ldots, \xbf_N,\sbf_1,\ldots, \sbf_N\vert A) = \pr... ... - \sum_i a_{ji} \sbf_{n,i}) \prod_j p_j(\sbf_{n,j})\right]. \end{displaymath}$ To do ML estimation of A we examine $\begin{displaymath}P(\xbf_1,\ldots, \xbf_N\vert A) = \prod_n P(\xbf_n\vert A), \end{displaymath}$ The ML solution is to find the A that maximizes this likelihood function. We have $\begin{displaymath}\begin{aligned} P(\xbf_n\vert A) &= \int P(\xbf_n\vert\sbf_... ...)} \prod_j p_j(\sum_i(A^{-1})_{ji}\xbf_{n,i}), \end{aligned} \end{displaymath}$ where the lemma we derived before has been used. The log likelihood function is $\begin{displaymath}\log P(\xbf_n\vert A) = -\log \det A + \sum_j \log p_j(\sum_i(A^{-1})_{ji}\xbf_{n,i}). \end{displaymath}$ Let W = A^-1. Then $\begin{displaymath}\log P(\xbf_n\vert A) = \log \det W + \sum_j \log p_j(\sum_i W_{ji}\xbf_{n,i}). \end{displaymath}$ Now we proceed as before, computing the derivative of the log likelihood with respect to W. The first part is easy: $\begin{displaymath}\partiald{}{W} \log \det W = W^{-T}. \end{displaymath}$ For the first part: $\begin{displaymath}\begin{aligned} \partiald{}{W_{mn}} \sum_j \log p_j(\sum_i W... ...frac{1}{p_m(a_m)} x_n \partiald{p_m(a_m)}{a_m} \end{aligned} \end{displaymath}$ Let $\begin{displaymath}\phi_m(a_m) = \frac{d}{d a_m} \log p_m(a_m) = \frac{1}{p_m(a_m)} \frac{d p_m(a_m)}{a_m} \end{displaymath}$ and let $z_m = \phi_m(a_m)$ . Then we have $\begin{displaymath}\partiald{}{W_{mn}}\sum_j \log p_j(\sum_i W_{ji} x_i) = x_n z_m. \end{displaymath}$ Thus $\begin{displaymath}\partiald{}{W} \log P(\xbf_n\vert A) = W^{-T} + \zbf \xbf^T. \end{displaymath}$ Compare with what we had before! Copyright 2008, by the Contributing Authors. Cite/attribute Resource. admin. (2006, May 17). Application of Information Theory to Blind Source Separation. Retrieved November 23, 2009, from Free Online Course Materials — USU OpenCourseWare Web site: http://ocw.usu.edu/Electrical_and_Computer_Engineering/Information_Theory/lecture4_3.htm. This work is licensed under a Creative Commons License.	Reuse Course Download this course

Sections

Personal tools

Application of Information Theory to Blind Source Separation

Document Actions

Mackay's approach