Personal tools
  •  

Markov Processes

Document Actions
  • Content View
  • Bookmarks
  • CourseFeed

Concepts  ::  Discrete  ::  Continuous  ::  States

Classes of States


\begin{definition}State $j$\ is {\bf accessible} from state $i$\ if $p_{ij}(n)...\par A Markov chain with a single class is {\bf irreducible}.\end{definition}


Example 6.3


Example 7


Example 8


Example 9


Definition 3

Let $f_i = P[\text{ever returning to state $i$}]$ . Then state $i$ is recurrent if $f_i = 1 .

If $f_i < 1 , then state $i$ is said to be transient .

  • If started in a transient state, then the state does not recur an infinite number of times.
  • If in a recurrent state, then the state recurs an infinite number of times.
Let $X_n$ denote the Markov chain with $X_o$ . Let

 \begin{displaymath} I_i(x) = \begin{cases}  1 \text{if }X=i \\ 0 \text{otherwise}\end{cases}.\end{displaymath}

Then

\begin{displaymath}E[\text{number of returns to state $i$}] = E\left[ \sum_{i}
  I_i(X_n)| X_0 = i\right] = \sum_{n=1}^\infty p_{ii}(n).\end{displaymath}

We see that recurrent means that  $\sum_{n=1}^\infty p_{ii}(n)=\infty$ .

Transient means that  $\sum_{n=1}^\infty p_{ii}(n) < \infty$ .

Example 10


Example 11


Example 12

Observation: The states of an irreducible, finite-state Markov chain are all recurrent.

Limiting probabilities

If all states are transient, then all the state probabilities approach 0 as $n \rightarrow \infty$ . If a M.C. has some transient classes and some recurrent classes, then eventually the process enters and remains in one of the recurrent classes. For limiting purposes, we can focus on individual recurrent classes.

Suppose a M.C. starts in a recurrent state $i$ at time 0. Let $T_i(1), T_i(1) + T_i(2), \ldots$ denote the times when the process returns to state $i$ , where T_i(k) is the time that elapses between the
Copyright 2008, by the Contributing Authors. Cite/attribute Resource . admin. (2006, June 08). Markov Processes. Retrieved January 07, 2011, from Free Online Course Materials — USU OpenCourseWare Web site: http://ocw.usu.edu/Electrical_and_Computer_Engineering/Stochastic_Processes/lecture10_4.htm. This work is licensed under a Creative Commons License Creative Commons License