Personal tools
You are here: Home → Electrical and Computer Engineering → Stochastic Processes → Homework Solutions

Homework Solutions

Document Actions
  • Send this
  • Print this
  • Content View
  • Bookmarks
  • CourseFeed

Utah State University
ECE 6010
Stochastic Processes
Homework # 4 Solutions

  1. Suppose $ \Xbf \sim \Nc(\mubf,\Sigma)$ .
    1. Show that $ E[\Xbf] = \mubf$ and $ \cov(\Xbf,\Xbf) = \Sigma$ .

      Using characteristic functions:

      $\displaystyle \phi_X(\ubf) = \exp[i \ubf^T \mubf - \frac{1}{2} \ubf^T \Sigma \ubf] $

      Taking the gradient with respect to $ \ubf$ we have

      $\displaystyle \partiald{\phi_X(\ubf)}{\ubf} = \exp[i \ubf^T \mubf - \frac{1}{2} \mubf^T \Sigma \mubf](i\mubf - \Sigma \ubf) $

      Now evaluating at $ \ubf = \zerobf$ and dividing by $ i$ we obtain

      $\displaystyle E[X] = \mubf. $

      Taking the gradient again with respect to $ \ubf$ we have

      $\displaystyle \exp[i \ubf^T \mubf - \frac{1}{2} \mubf^T \Sigma \mubf](-\Sigma) ... ...ac{1}{2} \mubf^T \Sigma \mubf](i \mubf - \Sigma \ubf)(i \mubf - \Sigma \ubf)^T $

      and evaluating at $ \ubf = \zerobf$ we have

      $\displaystyle -\Sigma - \mubf \mubf $

      Dividing by $ i^{-2}$ we obtain $ E[X^2] = \Sigma + \mubf\mubf^T$ , from which it follows that $ \cov(\Xbf,\Xbf) = \Sigma$ .

    2. Show that $ A \Xbf + \bbf \sim \Nc(A \mubf + \bbf, A \Sigma A^T)$ .

      We note that $ \phi_X(\ubf) = E[e^{i \ubf^T \Xbf}] = \exp[i \ubf^T \mubf - \frac{1}{2} \mubf^T \Sigma \mubf]$ . Then

      $\displaystyle \phi_Y(\ubf) = E[e^{i \ubf^T \Ybf}] = E[e^{i \ubf^T (A\Xbf + \bbf)}] = e^{i \ubf^T \bbf}E[e^{i \ubf^T A \Xbf}] $

      But we can compute the expected value using what we know from $ \phi_X(\ubf)$ :

      $\displaystyle E[e^{i \ubf^T A \Xbf}] = \left.\phi_X(\vbf)\right\vert _{\vbf = A^T \ubf} = \exp[i \ubf^T A \mubf - \frac{1}{2} \ubf^T A \Sigma A^T \ubf] $

      So

      $\displaystyle \phi_Y(\ubf) = e^{i \ubf^T \bbf}\exp[i \ubf^T A \mubf - \frac{1}{... ...A^T \ubf] = \exp[i\ubf(A\mubf+\bbf) - \frac{1}{2} \ubf^T (A \Sigma A^T) \ubf]. $

      From the form of the characteristic function we identify that $ \Ybf$ is Gaussian with

      $\displaystyle E[\Ybf] = A \mubf + \bbf \qquad \qquad \cov(\Ybf,\Ybf) = A\Sigma A^T. $

    3. Suppose $ \Sigma > 0$ 0$" align="middle" border="0" height="34" width="53" /> and write $ \Sigma = CC^T$ . Show that $ C^{-1}(\Xbf - \mubf) \sim \Nc(\zerobf,I)$ .

      If $ \Ybf = C^{-1}X - C^{-1}\mubf$ we have, by applying the previous result

      $\displaystyle E[\Ybf] = C^{-1} \mubf - C^{-1} \mubf = \zerobf $

      $\displaystyle \cov(\Ybf,\Ybf) = C^{-1} \Sigma C^{-T} = C^{-1} (CC^T) C^{-T} = I. $

  2. Suppose you have a random number generator which is capable of generating random numbers distributed as $ X \sim \Nc(0,1)$ . Describe how to generate random vectors $ \Ybf \sim \Nc(\mubf,\Sigma)$ .

    Generate $ n$ -dimensional random vectors $ \Xbf$ by generating $ n$ realizations of the scalar random variable $ X$ . Then $ \cov(\Xbf,\Xbf) = I$ .

    Write $ \Sigma = CC^T$ (the Cholesky factorization). Let $ \Ybf = C \Xbf + \mubf$ . Then

    $\displaystyle E[\Ybf] = \mbf $

    and

    $\displaystyle \cov(\Ybf,\Ybf) = C \cov(\Xbf,\Xbf) C^T = C I C^T = CC^T = \Sigma. $

  3. Suppose $ X$ and $ Y$ are r.v.s. Show that $ E[(X - h(Y))^2]$ is minimized over all functions $ h$ be the function

    $\displaystyle h(y) = E[X\vert Y=y]. $

    Assume $ E[X^2] < \infty$< \infty$" align="middle" border="0" height="38" width="99" /> .

    1. First approach: We will assume (only for convenience) that the r.v.s are continuous. We can write

      \begin{equation*}\begin{aligned} E[(X-h(Y))^2] &= E[ E[(X-Y(Y))^2\vert Y=y]] = ... ...X-h(y))^2 f_{X\vert Y}(X\vert Y=y) \,dx f_Y(y)\,dy \end{aligned}\end{equation*}

      To minimize this, we can minimize the inner integral

      $\displaystyle \int (X-h(y))^2 f_{X\vert Y}(X\vert Y=y) dx $

      for each value of $ h(y)$ . Since $ Y=y$ is a fixed value for the inner integral, $ h(y)$ is a fixed value for each value of $ y$ , and we can take the derivative with respect to $ h(y)$ and equate to zero to minimize:

      $\displaystyle \partiald{}{h(y)} \int (X-h(y))^2 f_{X\vert Y}(X\vert Y=y) dx = \int 2(X-h(y)) f_{X\vert Y}(X\vert Y=y) dx = 0 $

      which leads to

      $\displaystyle h(y) \int f_{X\vert Y}(X\vert Y=y) dx = \int X f_{X\vert Y}(X\vert Y=y) dx $

      The integral on the left is equal to 1 (since it integrates over the entire set of $ X$ ), and the integral on the right is equal to $ E[X\vert Y=y]$ .

    2. Approach 2: This one is more appealing, because it is not expressed in terms of integrals (making it immediately applicable to all types of distributions), it does not require interchanging limiting operations (i.e., taking derivatives inside integrals), and does not even require derivatives with respect to functions (which was only partially justified in the first approach). This presentation is due to Ross.

      The proof is accomplished by establishing the following inequality:

      $\displaystyle E[(X - h(Y))^2] \geq E[(X - E[X\vert Y])^2] $

      This is accomplished as follows:

      $\displaystyle \begin{aligned} E[(X-h(Y))^2\vert Y] &= E[(X - E[X\vert Y] + E[X\... ...ad 2 E[(X - E[X\vert Y])\underbrace{(E[X\vert Y] - h(Y))}\vert Y] \end{aligned}$

      as may be verified by expanding out the second line. However, given $ Y$ the term $ E[X\vert Y] - h(Y)$ (which has the underbrace), being a function of $ Y$ , can be treated as a constant and factored out of the expectation. Thus

      $\displaystyle \begin{aligned} E[(X - E[X\vert Y])(E[X\vert Y] - h(Y))\vert Y] &... ...Y))( E[X\vert Y] - E[X\vert Y]) \\ & = (E[X\vert Y] - h(Y))0 = 0 \end{aligned}$

      We thus obtain

      $\displaystyle \begin{aligned} E[(X-h(Y))^2\vert Y] &= E[(X - E[X\vert Y] + E[X\... ...&= E[(X-E[X\vert Y])^2\vert Y] + E[(E[X\vert Y] - h(Y))^2\vert Y] \end{aligned}$

      or

      $\displaystyle \begin{aligned} E[(X-h(Y))^2\vert Y] \geq E[(X-E[X\vert Y])^2\vert Y] \end{aligned}$

      Taking expectations of both sides we obtain

      $\displaystyle E[(X-h(Y))^2] \geq E[(X - E[X\vert Y])^2], $

      and equality holds if $ h(Y) = E[X\vert Y]$ .

  4. Let $ (X,Y) \sim \Nc(\mu_x,\mu_y,\sigma_x^2,\sigma_y^2,\rho)$ . Let $ \Xbf = \left[\begin{smallmatrix}X \\ Y\end{smallmatrix}\right] \sim \Nc(\mubf, \Sigma)$ , where

    $\displaystyle \Sigma = \begin{bmatrix}s_{11} & s_{12} s_{21} & s_{22} \end{bmatrix}. $

    Determine the relationship between $ \mu_x,\mu_y,\sigma_x^2,\sigma_y^2,\rho$ and $ \mubf$ and $ \Sigma$ .

    First observe that $ s_{21} = s_{12}$ . Write down two expressions for the density, first in terms of the separate parameters,

    $\displaystyle f_{XY}(x,y) = \frac{1}{2\pi \sigma_x \sigma_y \sqrt{1-\rho^2}} \e... ...x)(y-\mu_y)}{\sigma_x\sigma_y} + \frac{(y-\mu_y)}{\sigma_y^2} \right] \right\} $

    and second in terms of the matrix parameters,

    $\displaystyle f_\Xbf(\xbf) = \frac{1}{(2\pi)^{n/2}\vert\Sigma\vert^{1/2}} \exp\left[-\frac{1}{2}(\xbf - \mubf)^T \Sigma^{-1} (\xbf - \mubf)\right]. $

    Equating the constants in front we must have

    $\displaystyle \sigma_x^2 \sigma_y^2(1-\rho^2) = s_{11}s_{22} - s_{21}^2. $

    It is clear that $ \mu_x$ and $ \mu_y$ are the first and second elements of $ \mubf$ . Let us assume only for convenience that $ \mubf = \zerobf$ . Expanding the exponent of the second form of the pdf and equating to terms in the first form of the pdf we find

    $\displaystyle s_{11} - s_{21}^2/s_{22} = \sigma_x^2(1-\rho^2) $

    $\displaystyle s_{22} - s_{21}^2/s_{11} = \sigma_y^2(1-\rho^2) $

    $\displaystyle s_{11}s_{22}/s_{12} - s_{12} = \sigma_x\sigma_y/\rho $

    Comparing all of these, we have

    $\displaystyle \sigma_x^2 = s_{11} \qquad \qquad \sigma_y^2 = s_{22} \qquad \qquad \rho = \frac{s_{21}}{\sqrt{s_{11}s_{22}}} $

  5. Suppose $ \Xbf = \left[\begin{smallmatrix}X_1 X_2 X_3\end{smallmatrix}\right] \sim \Nc(\mubf,\Sigma)$ where

    $\displaystyle \mubf = \begin{bmatrix}1 2 3 \end{bmatrix}\qquad \qquad \Sigma = \begin{bmatrix}4 & 2 & 1 2 & 6 & 3 1 & 3 & 8 \end{bmatrix}$

    1. The value $ X_1 = 1.5$ is measured. Determine the best estimate for $ (X_2,X_3)$ .

      We can write $ \mubf_{2,3}$ for the mean of the unmeasured variables. The estimation formula is

      $\displaystyle \mubfhat_{2,3} = \mubf_{2,3} + \Sigma_{[2,3],1} \Sigma_{1}^{-1}(x_1 - \mu_1) $

      where $ \Sigma_{[2,3],1}$ is the covariance of the unmeasured values with the measured value,

      $\displaystyle \Sigma_{[2,3],1} = \begin{bmatrix} \cov(X_2,X_1) \\ \cov(X_3,X_1) \end{bmatrix} = \begin{bmatrix}2 1 \end{bmatrix}$

      and $ \Sigma_1 = \Sigma_{1,1} = 4$ . We obtain

      $\displaystyle \mubfhat_{2,} = \begin{bmatrix}2 3 \end{bmatrix} + \begin{bmat... ... \end{bmatrix} (4)^{-1}(1.5 - 1) = \begin{bmatrix}2.25 3.125 \end{bmatrix}. $

    2. In a separate problem, the values $ X_2 = 1$ and $ X_3 = 5$ are measured. Determine the best estimate of $ X_1$ .

      $\displaystyle \muhat_1 = \mu_1 + \Sigma_{1,[2,3]} \Sigma_{[2,3]}^{-1}(\begin{bmatrix}x_2 x_3 \end{bmatrix} - \mubf_{2,3}) $

      where

      $\displaystyle \Sigma_{1,[2,3]} = \begin{bmatrix}\cov(X_2,X_1) & \cov(X_3,X_1) \end{bmatrix} = \begin{bmatrix}2 & 1 \end{bmatrix}$

      and

      $\displaystyle \Sigma_{[2,3]} = \begin{bmatrix}6 & 3 3 & 8 \end{bmatrix}$

      Thus

      $\displaystyle \muhat_1 = 1 + \begin{bmatrix}2 & 1 \end{bmatrix}\begin{bmatrix}6... ...{bmatrix}1 5 \end{bmatrix} - \begin{bmatrix}2 3 \end{bmatrix}) = 0.6667. $

    3. Determine a random vector $ \Ybf$ which is a whitened version of $ \Xbf$ .

      Using MATLAB, 

      S = [4 2 1; 2 6 3; 1 3 8];
R = chol(S);
C = R';
C*C'   % check the result
      we determine that

      $\displaystyle \Sigma = \begin{bmatrix}2 & 0 & 0 1 & 2.2361 & 0 0.5 & 1.11... ...}2 & 0 & 0 1 & 2.2361 & 0 0.5 & 1.1180 & 2.5495 \end{bmatrix}^T = C C^T. $

      Now let $ \Ybf = C(\Xbf - \mubf)$ . Then $ \Ybf$ is whitened.
Copyright 2008, Todd Moon. Cite/attribute Resource. admin. (2006, June 13). Homework Solutions. Retrieved November 24, 2009, from Free Online Course Materials — USU OpenCourseWare Web site: http://ocw.usu.edu/Electrical_and_Computer_Engineering/Stochastic_Processes/hw4sol.html. This work is licensed under a Creative Commons License. Creative Commons License
Reuse Course
Download this course