Continuous-Time Markov Chains and Applications

I Prologue and Preliminaries.- 1 Introduction and Overview.- 1.1 Introduction.- 1.2 A Brief Survey.- 1.3 Outline of the Book.- 2 Mathematical Preliminaries.- 2.1 Introduction.- 2.2 Martingales.- 2.3 Markov Chains.- 2.4 Piecewise-Deterministic Processes.- 2.5 Irreducibility and Quasi-Stationary Distributions.- 2.6 Gaussian Processes and Diffusions.- 2.7 Notes.- 3 Markovian Models.- 3.1 Introduction.- 3.2 Birth and Death Processes.- 3.3 Finite-State Space Models.- 3.4 Stochastic Optimization Problems.- 3.5 Linear Systems with Jump Markov Disturbance.- 3.6 Interpretations of Time-Scale Separation.- 3.7 Notes.- II Singularly Perturbed Markov Chains.- 4 Asymptotic Expansion: Irreducible Generators.- 4.1 Introduction.- 4.2 Asymptotic Expansion.- 4.3 Regular Part of the Expansion.- 4.4 Boundary Layer Correction.- 4.5 Asymptotic Validation.- 4.6 Examples.- 4.7 Two-Time-Scale Expansion.- 4.8 Notes.- 5 Asymptotic Normality and Exponential Bounds.- 5.1 Introduction.- 5.2 Occupation Measure.- 5.3 Exponential Bounds.- 5.4 Asymptotic Normality.- 5.5 Extensions.- 5.6 Notes.- 6 Asymptotic Expansion: Weak and Strong Interactions.- 6.1 Introduction.- 6.2 Chains with Recurrent States.- 6.3 Inclusion of Absorbing States.- 6.4 Inclusion of Transient States.- 6.5 Countable-State Space - Part I.- 6.6 Countable-State Space - Part II.- 6.7 Remarks on Singularly Perturbed Diffusions.- 6.8 Notes.- 7 Weak and Strong Interactions: Asymptotic Properties and Ramification.- 7.1 Introduction.- 7.2 Aggregation of Markov Chains.- 7.3 Exponential Bounds.- 7.4 Asymptotic Normality.- 7.5 Measurable Generators.- 7.6 Notes.- III Control and Numerical Methods.- 8 Markov Decision Problems.- 8.1 Introduction.- 8.2 Problem Formulation.- 8.3 Limit Problem.- 8.4 Asymptotic Optimality.- 8.5 Convergence Rate and Error Bound.- 8.6 Long-Run Average Cost.- 8.7 Computational Procedures.- 8.8 Notes.- 9 Stochastic Control of Dynamical Systems.- 9.1 Introduction.- 9.2 Problem Formulation.- 9.3 Properties of the Value Functions.- 9.4 Asymptotic Optimal Controls.- 9.5 Convergence Rate.- 9.6 Weak Convergence Approach.- 9.7 Notes.- 10 Numerical Methods for Control and Optimization.- 10.1 Introduction.- 10.2 Numerical Methods for Optimal Control.- 10.3 Optimization under Threshold Policy.- 10.4 Notes.- A Appendix.- A.1 Properties of Generators.- A.2 Weak Convergence.- A.3 Relaxed Control.- A.4 Viscosity Solutions of HJB Equations.- A.5 Value Functions and Optimal Controls.- A.6 Miscellany.- References.