Personal tools
You are here: Home Electrical and Computer Engineering Signals and Systems Lecture 3: Informal Example

Lecture 3: Informal Example

Document Actions
  • Content View
  • Bookmarks
  • CourseFeed
Schedule :: Intro :: Informal Example :: Function Spaces

We briefly review the concept of a vector space. A vector space $V$ has the following key property: If $v, w \in V$ then $av+bw
\in V$ for any scalars $a$ and $b$ . That is, linear combinations of vectors give vectors.

Most of your background with vectors has been for vectors in $\Rbb^n$ . But: the signals that we deal with are also elements of a vector space , since linear combinations of signals also gives a signal. This is a very important and powerful idea.

Recall that in vector spaces we deal with concepts like the length of a vector, the angle between vectors, and the idea of orthogonal vectors. All of these concepts carry over, by suitable definitions, to vector spaces of signals.

This powerful idea captures most of the significant and interesting notions in signal processing, controls, and communications. This is really the reason why the study of linear algebra is so important.

In this lecture we will learn about geometric representations of signals via signal space (vector) concepts. This straightforward idea is the key to a variety of topics in signals and systems:

  1. It provides a distance concept useful in many pattern recognition techniques.
  2. It is used in statistical signal processing for the filtering, smoothing, and prediction of noisy signals.
  3. It forms the heart and geometric framework for the tremendous advances that have been made in digital communications.
  4. It is every waveform-based transform you ever wanted (Fourier series, FFT, DCT, wavelet, etc.)
  5. It is also used in the solution of partial differential equations, etc.
  6. It relies on our old friend, linearity. One might even say it is the reason that we care so much about linearity in the first place.

We will soon turn our attention to Fourier series, which are a way of analyzing and synthesizing signals.

Vectors will be written in bold font (like the ingredients above. Initially, we can think of a vector $\vbf$ as an ordered set of $n$ numbers, written in a column:

\begin{displaymath}\vbf = \left[\begin{array}{c} v_1  v_2  \vdots  v_n
\end{array}\right]. \end{displaymath}
Often to conserve writing, this will be written in transposed form,
\begin{displaymath}\vbf = [v_1,v_2,\ldots,v_n]^T.
Vector addition is component-by-component.

While we have written a vector as an $n$ -tuple, that is not what defines a vector. A vector is an element of a vector space, which is to say, it satisfies the linearity property given above.

Scalar multiplication of vectors is in the usual fashion. Matrix multiplication is also taken in the traditional manner.


\begin{displaymath}\gbf = [g_1,g_2,\ldots,g_n]^T
\begin{displaymath}\fbf = [f_1,f_2,\ldots,f_n]^T
be two vectors. The inner product (known to many of you as the dot product ) of the vectors $\gbf$ and $\fbf$ is written as
\begin{displaymath}\la \gbf,\fbf \ra = \gbf^T \fbf = \sum_{i=1}^n g_i f_i.
In words, multiply component by component, and add them up. Two vectors are said to be orthogonal or perpendicular if their inner product is zero:
\begin{displaymath}\la \fbf,\gbf \ra = 0 \quad \mbox{iff $\fbf$ and $\gbf$ are
If $\fbf$ and $\gbf$ are orthogonal, this is sometimes written $\fbf \perp \gbf.$

\begin{example}Let $\fbf = [1,-2,3]$ and $\gbf = [-3,3,3]$. Then
\begin{display... \ra = 0
\end{displaymath}so $\fbf$ and $\gbf$ are orthogonal.

The inner product can be expanded using the following rules:

  1. For a scalar $c$ ,
    \begin{displaymath}\la c \fbf,\gbf \ra = c \la \fbf,\gbf \ra \end{displaymath}
    \begin{displaymath}\la \fbf+\gbf,\hbf \ra = \la \fbf,\hbf \ra + \la \gbf,\hbf \ra \end{displaymath}
  3. For real vectors (which is all we will be concerned about for the moment)
    \begin{displaymath}\la \fbf,\gbf \ra = \la \gbf,\fbf \ra.

The (Euclidean) norm of a vector is given by

\begin{displaymath}\Vert\xbf\Vert = \la \xbf,\xbf \ra^{1/2} = \left(\sum_{i=1}^n
The distance between two vectors is given by
\begin{displaymath}d(\xbf,\ybf) = \Vert \xbf - \ybf\Vert = \left(\sum_{i=1}^n

The projection of a vector $\fbf$ onto a vector $\xbf$ is given by

\begin{displaymath}\proj(\fbf,\xbf) = \frac{\la \fbf,\xbf \ra}{\Vert \xbf \Vert^2 } =
\frac{\la \fbf,\xbf \ra}{\la \xbf,\xbf \ra }
Geometrically, this is the amount of the vector $\fbf$ in the direction of $\xbf$ . (Show a picture.) Obviously, if $\fbf$ and $\xbf$ are orthogonal, then the projection of $\fbf$ onto $\xbf$ is 0.

Now suppose that we have a vector $\ibf_1$ (an ``ingredient'') and we have a vector $\fbf$ and we want to make the best approximation to $\fbf$ using some amount of our ingredient. Draw a picture. We can write

\begin{displaymath}\fbf = c_1 \ibf_1 + \ebf_1
where $c_1$ is the amount of $\ibf_1$ we want and $\ebf_1$ is the error between the thing we want and our approximation of it. To get the best approximation we want to minimize the length of the error vector. Before we go through and do it the hard way, let us make a geometric observation. The length of the error is minimized when the error vector is orthogonal to our ingredient vector $\ibf_1$ :
\begin{displaymath}\ibf_1 \perp \ebf_1.
\begin{displaymath}(\fbf - c_1\ibf_1) \perp \ibf_1 \end{displaymath}

\begin{displaymath}\la (\fbf - c_1 \ibf_1),\ibf_1 \ra = 0 \end{displaymath}

\begin{displaymath}\la \fbf,\ibf_1 \ra = c_1 \la \ibf_1, \ibf_1 \ra
Giving us
\begin{displaymath}c_1 = \frac{\la \fbf,\ibf_1 \ra }
{ \la \ibf_1,\ibf_1 \ra } = \frac{\la \fbf,\ibf_1 \ra }{\Vert\ibf_1\Vert^2}
Note that this is simply the projection of $\fbf$ onto the vector $\ibf_1$ .

\begin{example}Suppose $\ibf = [1, 1]^T$ and $\fbf = [3,4]^T$. The
...T \ra} { \la [1,1]^T,[1,1]^T\ra } =

Now let's do it the hard way: we want to find the amount of $\ibf_1$ to minimize the (length of the) error. The squared length of the error is

\begin{displaymath}E = \Vert \ebf_1 \Vert^2 = \Vert \fbf - c_1 \ibf_1 \Vert^2 = ...
... \ra - 2c_1 \la \fbf,\ibf_1 \ra
+ c_1^2 \la \ibf_1,\ibf_1 \ra.
To minimize this, take the derivative with respect to the coefficient and equate to zero:
\begin{displaymath}\frac{dE}{dc_1} = -2 \la \fbf,\ibf_1 \ra + 2c_1 \la
\ibf_1,\ibf_1 \ra = 0
Solving for the coefficient,
\begin{displaymath}c_1 = \frac{\la \fbf,\ibf_1 \ra}{\la \ibf_1,\ibf_1 \ra}
This is the same one we got before.

We may actually have more than one ``ingredient'' vector to deal with. Suppose we want to approximate $\fbf$ with the vectors $\ibf_1$ and $\ibf_2$ . As before write

\begin{displaymath}\fbf = c_1 \ibf_1 + c_2 \ibf_2 + \ebf
where $\ebf$ is the error in the approximation. Note that we can write this in the following way:
\begin{displaymath}\fbf = \left[\begin{array}{cc} \ibf_1 & \ibf_2 \end{array}\right]
\left[\begin{array}{c} c_1  c_2 \end{array}\right] + \ebf.
using the usual matrix multiplication. We want to find the coefficients $c_1$ and $c_2$ to minimize the length of the error. We could to it the calculus way, or using our orthogonality idea. We will go for the latter: The error is orthogonal to the data means that
\begin{displaymath}\la \fbf - c_1 \ibf_1 - c_2\ibf_2,\ibf_1 \ra = 0

\begin{displaymath}\la \fbf - c_1 \ibf_1 - c_2\ibf_2,\ibf_2 \ra = 0 \end{displaymath}
(that is, the error is orthogonal to each of the ingredient "data'' points). Expanding these out gives
\begin{displaymath}\la \fbf,\ibf_1 \ra = c_1 \la \ibf_1,\ibf_1 \ra + c_2 \la
\ibf_2,\ibf_1 \ra \end{displaymath}

\begin{displaymath}\la \fbf,\ibf_2 \ra = c_1 \la \ibf_1,\ibf_2 \ra + c_2 \la
\ibf_2,\ibf_2 \ra \end{displaymath}
This is two equations in two unknowns that we can write in the form
\la \ibf_1,\ibf_1 \ra & \la \ibf_2,\i...
...\la \fbf,\ibf_1 \ra  \la \fbf,\ibf_2 \ra
If we know $\fbf$ and the ingredient vectors, we can solve for the coefficients.

\begin{example}Suppose $\fbf = [1,2,3]^T$ and $\ibf_1 = [1,1,0]^T$ and
...ctors $\ibf_1$ and
$\ibf_2$. The error in this case has length 3.

Of course, what we can do for two ingredient vectors, we can do for $n$ ingredient vectors (and $n$ may be infinite). We want to approximate $\fbf$ as

\begin{displaymath}\fbf = \sum_{i=1}^n c_i \ibf_i + \ebf
We can find the set of coefficients that minimize the length of the error $\ebf$ using the orthogonality principle as before, applied $n$ times. This gives us $n$ equations in the $n$ unknowns which may be written as
\la \ibf_1,\ibf_1 \ra & \la \ibf_2...
...ibf_2 \ra
 \vdots  \la \fbf,\ibf_n \ra
This could be readily solved (say, using Matlab).

It would seem that if we take $n$ large enough, we should be able to represent any vector. without any error. (Analogy: given enough ingredients, we could make any cake. We might not be able to make everything, but we could make everything some class of objects.) If this is true, the set of ingredient vectors are said to be complete . A more formal name for the ingredient vectors is basis vectors .

Although we have come up with a way of doing the approximation, there is still a lot of work to solve for the coefficients, since we have to first find a matrix and then invert it. Something that is commonly done is to choose a set of basis vectors that is orthogonal . That is, if $\ibf_j$ and $\ibf_k$ are any pair of basis vectors, then

\begin{displaymath}\la \ibf_j,\ibf_k \ra = 0 \quad\mbox{if } j\neq k
Let us return to the case of two basis vectors when the vectors are orthogonal. Then the equation for the coefficients becomes

\la \ibf_1,\ibf_1 \ra & 0 \\
0 & \la...
...\la \fbf,\ibf_1 \ra  \la \fbf,\ibf_2 \ra
\begin{displaymath}c_1 \la \ibf_1,\ibf_1 \ra = \la \fbf,\ibf_1 \ra

\begin{displaymath}c_2 \la \ibf_2,\ibf_2 \ra = \la \fbf,\ibf_2 \ra
so the coefficients are
\begin{displaymath}c_1 = \frac{\la \fbf,\ibf_1 \ra}{\la \ibf_1,\ibf_1 \ra} \quad\quad
c_2 = \frac{\la \fbf,\ibf_2 \ra}{\la \ibf_2,\ibf_2 \ra}
So solving for the coefficients in this case is as easy as doing it for the case of a single vector, and the coefficient is simply the projection of $\fbf$ onto its corresponding basis vector. This generalizes to $n$ basis vectors: If the basis vectors are orthogonal, then the coefficient is simply

\begin{displaymath}c_n = \frac{\la \fbf,\ibf_n\ra}{\la \ibf_n,\ibf_n \ra}.

\begin{example}Let us repeat the previous example, then $\fbf = [1,2,3]^T$,
...computations are
easier because the ingredient vectors are easier.

Copyright 2008, by the Contributing Authors. Cite/attribute Resource . admin. (2006, May 22). Lecture 3: Informal Example. Retrieved January 07, 2011, from Free Online Course Materials — USU OpenCourseWare Web site: This work is licensed under a Creative Commons License Creative Commons License