Application of Information Theory to Blind Source Separation

Document Actions

Introduction :: BSS :: Mackay's Approach :: Natural Gradient :: p(u)

So what about p(u)

We have observed that if $p(u) = \vert\partiald{g}{u}\vert$ , then the last terms of the density go away. We also commented above that the given form works only for super-Gaussian sources. So what do we do in the case where this is not a good assumption. One approach is to deal with two densities. On the one hand, we take

$\begin{displaymath}p(u) = \frac{1}{2}(N(\mu,\sigma^2) + N(-\mu,\sigma^2)) \end{displaymath}$

giving (where $a = \mu/\sigma^2)$

$\begin{displaymath}\phi(u) = \frac{u}{\sigma^2} - a\left(\frac{\exp(au) - \exp(-... ...\right) = \mu/\sigma^2 - \mu/\sigma^2 \tanh(\mu u/\sigma^2). \end{displaymath}$

This p is subGaussian. When $\mu=1$ and $\sigma^2$ = 1 we obtain

$\begin{displaymath}\phi(u) = - \tanh(u), \end{displaymath}$

and the learning rule (employing natural gradient) is

$\begin{displaymath}\Delta W \propto [I + \tanh(\ubf)\ubf^T - \ubf\ubf^T]W. \end{displaymath}$

The superGaussian density is modeled as

$\begin{displaymath}p(u) \propto [\Nc(0,1)] \sech^2(u) \end{displaymath}$

giving rise to

$\begin{displaymath}\phi(u) = u + \tanh(u) \end{displaymath}$

and the learning rule

$\begin{displaymath}\Delta W \propto [I - \tanh(\ubf)\ubf^T - \ubf\ubf^T]W. \end{displaymath}$

Combining these, we obtain

$\begin{displaymath}\Delta W \propto \begin{cases} (I - \tanh(\ubf)\ubf^T - \... ...bf^T - \ubf\ubf^T) W & \text{sub-Gaussian} \\ \end{cases} \end{displaymath}$

The decision between rules can be made on an element-by-element basis. A decision must be based on the available data.

Copyright 2008, by the Contributing Authors. Cite/attribute Resource. admin. (2006, May 17). Application of Information Theory to Blind Source Separation. Retrieved November 23, 2009, from Free Online Course Materials — USU OpenCourseWare Web site: http://ocw.usu.edu/Electrical_and_Computer_Engineering/Information_Theory/lecture4_5.htm. This work is licensed under a Creative Commons License.

Course Contents Information Theory Home About the Professor Syllabus Schedule Homework Assignments Programming Assignments Download This Course	Personal tools Log in You are here: Home → Electrical and Computer Engineering → Information Theory → Application of Information Theory to Blind Source Separation
	Info Application of Information Theory to Blind Source Separation Document Actions Introduction :: BSS :: Mackay's Approach :: Natural Gradient :: p(u) So what about p(u) We have observed that if $p(u) = \vert\partiald{g}{u}\vert$ , then the last terms of the density go away. We also commented above that the given form works only for super-Gaussian sources. So what do we do in the case where this is not a good assumption. One approach is to deal with two densities. On the one hand, we take $\begin{displaymath}p(u) = \frac{1}{2}(N(\mu,\sigma^2) + N(-\mu,\sigma^2)) \end{displaymath}$ giving (where $a = \mu/\sigma^2)$ $\begin{displaymath}\phi(u) = \frac{u}{\sigma^2} - a\left(\frac{\exp(au) - \exp(-... ...\right) = \mu/\sigma^2 - \mu/\sigma^2 \tanh(\mu u/\sigma^2). \end{displaymath}$ This p is subGaussian. When $\mu=1$ and $\sigma^2$ = 1 we obtain $\begin{displaymath}\phi(u) = - \tanh(u), \end{displaymath}$ and the learning rule (employing natural gradient) is $\begin{displaymath}\Delta W \propto [I + \tanh(\ubf)\ubf^T - \ubf\ubf^T]W. \end{displaymath}$ The superGaussian density is modeled as $\begin{displaymath}p(u) \propto [\Nc(0,1)] \sech^2(u) \end{displaymath}$ giving rise to $\begin{displaymath}\phi(u) = u + \tanh(u) \end{displaymath}$ and the learning rule $\begin{displaymath}\Delta W \propto [I - \tanh(\ubf)\ubf^T - \ubf\ubf^T]W. \end{displaymath}$ Combining these, we obtain $\begin{displaymath}\Delta W \propto \begin{cases} (I - \tanh(\ubf)\ubf^T - \... ...bf^T - \ubf\ubf^T) W & \text{sub-Gaussian} \\ \end{cases} \end{displaymath}$ The decision between rules can be made on an element-by-element basis. A decision must be based on the available data. Copyright 2008, by the Contributing Authors. Cite/attribute Resource. admin. (2006, May 17). Application of Information Theory to Blind Source Separation. Retrieved November 23, 2009, from Free Online Course Materials — USU OpenCourseWare Web site: http://ocw.usu.edu/Electrical_and_Computer_Engineering/Information_Theory/lecture4_5.htm. This work is licensed under a Creative Commons License.	Reuse Course Download this course

Sections

Personal tools

Application of Information Theory to Blind Source Separation

Document Actions

So what about p(u)