Personal tools
You are here: Home → Electrical and Computer Engineering → Stochastic Processes → Homework Solutions

Homework Solutions

Document Actions
  • Send this
  • Print this
  • Content View
  • Bookmarks
  • CourseFeed

Utah State University
ECE 6010
Stochastic Processes
Homework # 11 Solutions

  1. Let $ M_{n}$ denote the sequence of sample means from an iid random process $ X_{n}$ :

    $\displaystyle M_{n} = \frac{X_1 + X_2 + \cdots + X_n }{n}.$

    1. Is $ M_n$ a Markov process?
      $\displaystyle M_{n}$ $\displaystyle =$ $\displaystyle \frac{1}{n} \sum_{i=1}^{n} X_{i} = \frac{1}{n}[X_{n} + (n-1) M_{n-1}]$  
        $\displaystyle =$ $\displaystyle \frac{1}{n} X_{n} + (1- \frac{1}{n}) M_{n-1}$  

      Clearly if $ M_{n-1}$ is given then $ M_{n}$ depends only on $ X_{n}$ and is independent of $ M_{n-2}, M_{n-3}, \ldots$ . Therefore, $ M_{n}$ is a Markov process.

    2. If the answer to part a is yes, find the following state transition pdf: $ f_{M_{n}}(x\vert M_{n-1} = y)$ .
      $\displaystyle f_{M_{n}}(x\vert M_{n-1} = y)$ $\displaystyle =$ $\displaystyle P[ x < M_{n} \leq x + dx \vert M_{n-1} = y]$< M_{n} \leq x + dx \vert M_{n-1} = y]$" align="middle" border="0" height="31" width="231" />  
        $\displaystyle =$ $\displaystyle P[ x < \frac{1}{n} + (1-\frac{1}{n}) y \leq x+ dx]$< \frac{1}{n} + (1-\frac{1}{n}) y \leq x+ dx]$" align="middle" border="0" height="49" width="216" />  
        $\displaystyle =$ $\displaystyle P[nx - (n-1)y < x_{n} \leq nx - (n-1)y + dx]$< x_{n} \leq nx - (n-1)y + dx]$" align="middle" border="0" height="31" width="313" />  
        $\displaystyle =$ $\displaystyle f_{x}(nx - (n-1)y)dx$  

  2. An urn initially contains five black balls and five white balls. The following experiment is repeated indefinitely: A ball is drawn from the urn; if the ball is white it is put back in the urn, otherwise it is left out. Let $ X_{n}$ be the number of black balls remaining in the urn after $ n$ draws from the urn.
    1. Is $ X_{n}$ a Markov process? If so, find the apropriate transition probabilities.
    2. Do the transition probabilities depend on $ n$ ?

    The number $ X_{n}$ of black balls in the urn completely specifies the probability of the outcomes of a trial; therefore $ X_{n}$ is independent of its past values and $ X_{n}$ is a Markov proces.

    $\displaystyle P[X_{n} = 4 \vert X_{n-1} = 5]$ $\displaystyle =$ $\displaystyle \frac{5}{10} = 1 - P[X_{n} = 5 \vert X_{n-1} = 5]$  
    $\displaystyle P[X_{n} = 3 \vert X_{n-1} = 4]$ $\displaystyle =$ $\displaystyle \frac{4}{9} = 1 - P[X_{n} = 4 \vert X_{n-1} = 4]$  
    $\displaystyle P[X_{n} = 2 \vert X_{n-1} = 3]$ $\displaystyle =$ $\displaystyle \frac{3}{8} = 1 - P[X_{n} = 3 \vert X_{n-1} = 3]$  
    $\displaystyle P[X_{n} = 1 \vert X_{n-1} = 2]$ $\displaystyle =$ $\displaystyle \frac{2}{7} = 1 - P[X_{n} = 2 \vert X_{n-1} = 2]$  
    $\displaystyle P[X_{n} = 0 \vert X_{n-1} = 1]$ $\displaystyle =$ $\displaystyle \frac{1}{6} = 1 - P[X_{n} = 1 \vert X_{n-1} = 1]$  
    $\displaystyle P[X_{n} = 0 \vert X_{n-1} = 0]$ $\displaystyle =$ $\displaystyle 1$  

    All the transition probability are independent of time.

  3. Let $ X_{n}$ be the Bernoulli iid process, and let $ Y_n$ be given by $ Y_n = X_n +X_{n-1}$ . It was shown in Example 8.2 that $ y_n$ is not a Markov process. Consider the vector process defined by $ \mathbf{Z}_n = (X_n ,X_{n-1})$ .
    1. Show that $ \mathbf{Z}_n$ is a Markov process.

      $\displaystyle P[\mathbf{Z}_{n+1} = (x_{n+1},x_n)\vert \underbrace{\mathbf{Z}_{n... ...n-1}), \mathbf{Z}_{n-1} = (x_{n-1},x_{n-2}),\ldots}_{\mbox{all past vectors}}] $


      $\displaystyle =$   $\displaystyle P[\mathbf{Z}_{n+1} = (x_{n+1},x_n)\vert\underbrace{X_{n} = x_{n},X_{n-1} \ldots}_{\mbox{all past Bernoulli trials}}]$  
      $\displaystyle =$   $\displaystyle P[\underbrace{X_{n+1} = x_{n+1}}_{\mbox{next trial}}]$  
      $\displaystyle =$ $\displaystyle P[\mathbf{Z}_{n+1} = (x_{n+1},x_{n}) \vert \mathbf{Z}_{n} = (x_{n},x_{n-1}) ]$    

      Therefore, $ \mathbf{Z}_{n} $ is a Markov process.

    2. Find the state transition diagram for $ \mathbf{Z}_n$ .
      Figure: State transition diagram for $ \mathbf{Z}_n$
      \includegraphics{transprob.eps}

      where $ p = P[x=1]$ .

  4. Show that the following autoregressive process is a Markov process: $ Y_n = rY_{n-1}+X_{n}$ where $ Y_{0} = 0$ and $ X_{n}$ is an iid process.

    $\displaystyle Y_{n} = rY_{n-1} + X_{n} \quad \quad Y_{0} = 0 \quad \quad Y_{n} - rY_{n-1} = X_{n}$


    $\displaystyle f_{Y_{n}}(y\vert Y_{n-1} = y_{1} Y_{n-2} = y_{n-2} \ldots)$ $\displaystyle =$ $\displaystyle f_{X_{n}}(y-ry_{1})$  
      $\displaystyle =$ $\displaystyle f_{Y_{n}}(y\vert Y_{n-1} = y_{1})$  

    $ \Rightarrow Y_{n}$ is a Markov process.
  5. Let $ X_{n}$ be the Markov chain defined in Problem 2.
    1. Find the one-step transition probability matrix $ P$ for $ X_{n}$ .

      $\displaystyle P = \left[ \begin{array}{cccccc} 1 & 0 & 0 & 0 & 0 & 0 \\ \frac{... ...9} & 0 \\ 0 & 0 & 0 & 0 & \frac{5}{10} & \frac{5}{10} \\ \end{array} \right] $

    2. Find the two-step transition probability matrix $ P^2$ by matrix multiplication. Check your answer by computing $ p_{54}(2)$ and comparing it to the correspoinding entry in $ P^2$ .

      $\displaystyle P^2 = \left[ \begin{array}{cccccc} 1 & 0 & 0 & 0 & 0 & 0 \\ \fra... ... 0 & 0 & 0 & \frac{2}{9} & \frac{19}{36} & \frac{1}{4} \\ \end{array} \right] $

      $\displaystyle p_{54}(2) = p_{55}^{no change}(1) p_{54}^{no change}(1) + p_{... ... \frac{1}{2} \cdot \frac{1}{2} + \frac{1}{2} \cdot \frac{5}{9} = \frac{19}{36} $

      From the $ P^2$ matrix $ p_{54}(2) = \frac{19}{36}$ .
    3. What happens to $ X_{n}$ as $ n$ approaches infinity ? Use your answer to guess the limit of $ P^n$ as $ n \rightarrow \infty$ .
      As $ n \rightarrow \infty$ eventually all black balls are removed. Thus

      $\displaystyle P^n = \left[ \begin{array}{cccccc} 1 & 0 & 0 & 0 & 0 & 0 \\ 1 &... ... 0 \\ 1 & 0 & 0 & 0 & 0 & 0 \\ 1 & 0 & 0 & 0 & 0 & 0 \\ \end{array} \right] $

  6. Two gambler play the following game. A fair coin is flipped; if the outcome is heads, player $ A$ pays player $ B$ $ 1, and if the outcoime is tails player $ B$ plays player $ A$ $ 1. the game is continued until one of the players goes broke. Suppose that initially player $ A$ has $ 1 and player $ B$ has $ 2, so a total of $ 3 is up for grabs. Let $ X_n$ denote the number of dolars held by player $ A$ after $ n$ trials.
    1. Show that $ X_n$ is a Markov chain.
      $ X_{n} \in 0,1,2,3$
      $ P[X_n = k\vert X_{n-1} = j,\ldots] = P[X_{n} = k \vert X_{n-1}=j]$
      since $ X_{n} = X_{n-1} \pm 1$ for $ X_{n-1} \in \{1,2\}$
      and $ X_{n} = X_{n-1}$ if $ X_{n-1} \in \{0,3\}$ .

    2. Sketch the state transition diagram for $ X_n$ and give the one-step transition probability matrix $ P$ .
      Figure 2: State transition diagram for $ X_n$ .
      \includegraphics{trans1.eps}

      $\displaystyle P = \left[ \begin{array}{cccc} 1 & 0 & 0 & 0\\ \frac{1}{2} & 0 ... ... 0 \\ 0 & \frac{1}{2} & 0 & \frac{1}{2} \\ 0 & 0 & 0 & 1 \end{array} \right] $

    3. Use the state trasition diagram to help you show that for $ n$ even $ p_{ii}(n) = (1/2)^n$ for $ i = 1,2$ and $ p_{10}(n) = 2/3(1-(1/4)^k) = p_{23}(n)$ .

      For $ n=2k, i \in \{1,2\}$

      $\displaystyle p_{ii}(n)$ $\displaystyle =$ $\displaystyle P \overbrace{[HT HT HT \ldots HT]}^{2k} =\left(\frac{1}{2}\right)^n$  
      $\displaystyle p_{10}(n)$ $\displaystyle =$ $\displaystyle \sum_{j=0}^{k-1} P[j 1 \rightarrow $   2 cycles and then go to 0$\displaystyle ]$  
        $\displaystyle =$ $\displaystyle \sum_{j=0}^{k-1} \left(\frac{1}{2}\right)^{2j} \frac{1}{2}$  
        $\displaystyle =$ $\displaystyle \frac{2}{3} \left( 1 - \left( \frac{1}{4} \right)^k \right)$  
      $\displaystyle p_{23}(n)$ $\displaystyle =$ $\displaystyle p_{10}(n)$   by symmetry.  

    4. Find the $ n$ -step transition probability matrix for $ n$ even using part c.

      $\displaystyle P(n) = \left[ \begin{array}{cccc} 1 & 0 & 0 & 0 \\ \frac{2}{3} ... ... 1 - \left( \frac{1}{4} \right)^k \right)\\ 0 & 0 & 0 & 1 \end{array} \right] $

    5. Find the limit of $ P^{n}$ as $ n \rightarrow \infty$ .

      $\displaystyle P(n) \rightarrow \left[ \begin{array}{cccc} 1 & 0 & 0 & 0\\ \fr... ...} \\ \frac{1}{3} & 0 & 0 & \frac{2}{3} \\ 0 & 0 & 0 & 1 \end{array} \right] $

    6. Find the probability that player $ A$ eventually wins.
      $\displaystyle \mathbf{p}(n)$ $\displaystyle =$ $\displaystyle [ 0 1 0 0 ] P(n)$  
        $\displaystyle =$ $\displaystyle \left[ \frac{2}{3} \left( 1 - \left( \frac{1}{4} \right)^k \right... ...ht)^k , 0 , \frac{1}{3} \left( 1 - \left( \frac{1}{4} \right)^k \right) \right]$  
        $\displaystyle \rightarrow$ $\displaystyle \left[ \frac{2}{3}, 0, 0, \frac{1}{3} \right]$  

      $ P$ [player $ A$ wins] = $ \frac{1}{3}$ .

  7. A machine consists of two parts that fail and are repaired independently. A working part fails during any given day with probability $ a$ . A part that is not working is repaired by the next day with probability $ b$ . Let $ X_{n}$ be the number of working parts in day $ n$ .
    1. Show that $ x_{n}$ is a three-state Markov chain and give its one-step transition probability matrix $ P$ .

      $ X_{n} \in \{0,1,2\}$

      \begin{figure}\centering % \input{trans2.pstex_t} \end{figure}

      $\displaystyle P = \left[ \begin{array}{ccc} (1-b)^2 & 2b(1-b) & b^2 \\ a(1-b) & (1-a)(1-b) + ab & b(1-b) \\ a^2 & 2a(1-a) & (1-a)^2 \end{array} \right] $

    2. Show that the steady state pmf $ \underline{\pi}$ is binomial with parameter $ p = b/(a+b)$ .

      Claim: the steady state pmf is

      $\displaystyle \underline{\pi}$ $\displaystyle =$ $\displaystyle \left[ \left( \frac{a}{a+b} \right)^2 , 2 \left( \frac{b}{a+b} \right)\left( \frac{a}{a+b} \right) , \left( \frac{b}{a+b} \right)^2 \right]$  
        $\displaystyle =$ $\displaystyle \frac{1}{(a+b)^2} ( a^2, 2ab,b^2)$  


      $\displaystyle \underline{\pi} P$ $\displaystyle =$ \begin{displaymath}\frac{1}{(a+b)^2} ( a^2, 2ab,b^2) \left[ \begin{array}{ccc} (... ...) + ab & b(1-a) \\ a^2 & 2a(1-a) & (1-a)^2 \end{array} \right]\end{displaymath}  
        $\displaystyle =$ \begin{displaymath}\frac{1}{(a+b)^2} \left[ \begin{array}{c} a^2(1-b)^2 + 2a^2 b... ...1-a)\\ a^2 b^2 + 2ab^2 (1-a) + b^2 (1-a)^2 \end{array} \right]\end{displaymath}  
        $\displaystyle =$ $\displaystyle \frac{1}{(a+b)^2} ( a^2, 2ab,b^2)$  
        $\displaystyle =$ $\displaystyle \underline{\pi}$  

    3. What do you expect is steady state pmf for a machine that consists of $ n$ parts?

      $\displaystyle P[X_{n} = k] = \left( \begin{array}{c} n \\ k \end{array} \righ... ...a}{a+b}\right)^k \left( \frac{b}{a+b} \right)^{n-k} \quad \quad 0\leq k \leq n $

Copyright 2008, Todd Moon. Cite/attribute Resource. admin. (2006, June 13). Homework Solutions. Retrieved November 23, 2009, from Free Online Course Materials — USU OpenCourseWare Web site: http://ocw.usu.edu/Electrical_and_Computer_Engineering/Stochastic_Processes/hw11sol.html. This work is licensed under a Creative Commons License. Creative Commons License
Reuse Course
Download this course