Personal tools

Random Vectors

Document Actions
  • Content View
  • Bookmarks
  • CourseFeed

Vectors  ::  Covariance  ::  Functions  ::  Application  ::  Markov Model

An Application: MMSE Prediction

Suppose we have a random sequence $X_1, X_2, \ldots, X_n$ , and we observe the first $n-1$ of them:

\begin{displaymath}X_1 = x_1, X_2=x_2, \ldots, X_{n-1} = x_{n-1}.

Given this data, we want to predict the value of $X_n$ .

Our estimate of $x_n$ will be denoted as $\xhat_n$ . Clearly, it could be a function of all the observed data:

\begin{displaymath}\xhat_n = h(x_1,x_2,\ldots, x_{n-1})

for some function $h : \Rbb^{n-1} \rightarrow \Rbb$ . One thing we could try is to minimize the average of $(x_n - h(x_1,\ldots,
x_{n-1}))^2$ . That is we would like to solve

\begin{displaymath}\min_{h} E[(X_n - h(x_1,x_2,\ldots, x_{n-1}))^2]

It is easy to see (HW!) that the best such $h$ is

\begin{displaymath}h(x_1,x_2,\ldots, x_n) = E[X_n\vert X_1 = x_1, X_2=x_2, \ldots, X_{n-1}
= x_{n-1}].

That is, the best estimator (in a minimum mean-squared error sense) is the conditional expectation!

Now let us take a specific distribution. Suppose $\Xbf \sim \Nc(\mubf,\Sigma)$ , and partition according to

\begin{displaymath}\begin{bmatrix}X_1  X_2  \vdots  X_{n-1}  X_n

Given $X_1 = x_1, \ldots, X_{n-1}=x_{n-1}$ , the variable $X_n$ is $\Nc(\mu', \sigma^2)$ where

\begin{displaymath}\mu' = \mu_n + \Sigma_{n,n-1}\Sigma_{n-1}^{-1}\begin{bmatrix}...
...  x_2 - \mu_2  \vdots  x_{n-1} - \mu_{n-1}

\begin{displaymath}\sigma^2 = \sigma_n^2 - \Sigma_{n,n-1} \Sigma_{n-1}^{-1}


\begin{displaymath}\Sigma = \begin{bmatrix}\Sigma_{n-1} & \Sigma_{n-1,n} \\
\Sigma_{n,n-1} & \sigma_n^2


\begin{displaymath}\Sigma_{n-1} = \cov([X_1,\ldots,X_{n-1}], [ ])

\begin{displaymath}\Sigma_{n,n-1} = (\cov(X_n,X_1), \cov(X_n,X_1), \ldots,

So $\mu'$ is the conditional mean that we want and $\sigma^2$ is the variance of the conditional distribution,

\begin{displaymath}\sigma^2 = \var(X_n\vert X_1,\ldots, X_{n-1}) = E[(X_n -
\mu')^2\vert X_1=x_1,\ldots, X_{n-1} = x_{n-1}].

This is the minimum mean-squared error (MMSE).

Notationally, write

\begin{displaymath}\abf^T = \Sigma_{n,n-1} \Sigma_{n-1}^{-1}.


\begin{displaymath}\xhat_n = \mu_n + \abf^T \begin{bmatrix}x_1 - \mu_1  \vdots \\
x_{n-1} - \mu_{n-1} \end{bmatrix}\end{displaymath}

This is just a digital filter!

We can also show that $\sigma^2 \leq \sigma_n^2$ , so that incorporating information from measurements decreases our uncertainty.

Note: For a Gaussian r.v., the MMSE estimator is linear .

Copyright 2008, by the Contributing Authors. Cite/attribute Resource . admin. (2006, May 31). Random Vectors. Retrieved January 07, 2011, from Free Online Course Materials — USU OpenCourseWare Web site: This work is licensed under a Creative Commons License Creative Commons License