### CS558, Homework 1

• Question 1 (30 pts)

• Consider a Markov Chain with the following transition probability matrix:

 ``` 1 2 3 4 +- -+ | 1/4 1/4 1/4 1/4 | 1 | 0 0 1 0 | 2 P = | 0 0 0 1 | 3 | 1 0 0 0 | 4 +- -+ ```

Questions:

 Draw the state transition diagram (the states are numbered as 1, 2, 3 and 4). (10 pts) Given that the system is current in state 1, what is the probability that it will be state 3 in 4 transitions ? (10 pts) What is the limiting (steady) state probablities π1, π2, π3 and π4. (10 pts)

• Question 2 (30 pts)

• A professor for a course left 3 books - labeled 1, 2 and 3 - in the reference section of the library. Students are not allowed to leave the library with these books.

However, Bob has discovered a way to beat the system:

 If he comes just before the library closes, then he can check out one of the books for the night and return it the following morning. He is only allowed to check out one book at a time.

Bob's habit for checking out the books can be characterized as follows:

 Given that he checks out book 1 today, he is equally likely to check out books 1 or 2 tomorrow. Given that he checks out book 2 today, tomorrow he will check out book 1 with probability 0.5, book 2 with probability 0.25 and book 3 with probability 0.25. Given that he checks out book 3 today, tomorrow he will check out book 2 with probability 0.25 and book 3 with probability 0.75.

Questions:

 Draw the state transition diagram (the states are to be numbered as 1, 2 and 3). (5 pts) Give the one-step probability matrix (5 pts) Given that Bob has checked out book 1 today, what is the probability that he will check out book 3 in 4 days time ? (10 pts) Find the limiting (steady) state probabilities π1, π2 and π3. (10 pts)

• Question 3 (40 pts)

• Consider the following discrete-time Markov chain:

Questions:

• List all: (10 pts)

 transient states periodic states and their periods recurrent states chains and the states in each chain

• Given that the process is currently in state 3, what is the probability that it will never enter state 1 (10 pts)

• Give the one-step transition probability matrix (10 pts)

• Given that the process starts in state 4. We let the process run for a very long time (say 1,000,000 steps) and then make an observation.

What is the probability that the process is in state 6 when we observe it ? (10 pts)