mdp

This module has been merged in QuantEcon.py (version 0.2.0 or above) as DiscreteDP.

To try, type

pip install quantecon

at a terminal prompt.

Quick Start

from quantecon.markov import DiscreteDP

Creating a DiscreteDP instance

Product formulation

>>> R = [[5, 10], [-1, -float('inf')]]
>>> Q = [[(0.5, 0.5), (0, 1)], [(0, 1), (0.5, 0.5)]]
>>> beta = 0.95
>>> ddp = DiscreteDP(R, Q, beta)

State-action pairs formulation

>>> s_indices = [0, 0, 1]  # State indices
>>> a_indices = [0, 1, 0]  # Action indices
>>> R = [5, 10, -1]
>>> Q = [(0.5, 0.5), (0, 1), (0, 1)]
>>> beta = 0.95
>>> ddp = DiscreteDP(R, Q, beta, s_indices, a_indices)

Solving the model

Policy iteration

>>> res = ddp.solve(method='policy_iteration', v_init=[0, 0])
>>> res.sigma  # Optimal policy function
array([0, 0])
>>> res.v  # Optimal value function
array([ -8.57142857, -20.        ])
>>> res.num_iter  # Number of iterations
2

Value iteration

>>> res = ddp.solve(method='value_iteration', v_init=[0, 0],
...                 epsilon=0.01)
>>> res.sigma  # (Approximate) optimal policy function
array([0, 0])
>>> res.v  # (Approximate) optimal value function
array([ -8.5665053 , -19.99507673])
>>> res.num_iter  # Number of iterations
162

Modified policy iteration

>>> res = ddp.solve(method='modified_policy_iteration',
...                 v_init=[0, 0], epsilon=0.01)
>>> res.sigma  # (Approximate) optimal policy function
array([0, 0])
>>> res.v  # (Approximate) optimal value function
array([ -8.57142826, -19.99999965])
>>> res.num_iter  # Number of iterations
3

Lecture in quant-econ.net

• Discrete Dynamic Programming

Notebooks

• Getting started
• Implementation details
• Examples
  • Automobile replacement (Rust 1996)
  • Optimal growth
  • Job search
  • Career choice
  • Asset replacement (Miranda and Fackler Section 7.6.2)
  • Asset replacement with maintenance (Miranda and Fackler Section 7.6.3)
  • Option pricing (Miranda and Fackler Section 7.6.4)
  • Water management (Miranda and Fackler Section 7.6.5)
  • POMDP Tiger example
• Perfomance comparison
  • Machine 1
  • Machine 2