Stochastic control

A typical specification of the discrete-time stochastic linear quadratic control problem is to minimize[2]^{: ch. 13,}[3][5]

${\displaystyle \mathrm {E} _{1}\sum _{t=1}^{S}\left[y_{t}^{\mathsf {T}}Qy_{t}+u_{t}^{\mathsf {T}}Ru_{t}\right]}$

where E₁ is the expected value operator conditional on y₀, superscript T indicates a matrix transpose, and S is the time horizon, subject to the state equation

${\displaystyle y_{t}=A_{t}y_{t-1}+B_{t}u_{t},}$

where y is an n × 1 vector of observable state variables, u is a k × 1 vector of control variables, A_t is the time t realization of the stochastic n × n state transition matrix, B_t is the time t realization of the stochastic n × k matrix of control multipliers, and Q (n × n) and R (k × k) are known symmetric positive definite cost matrices. We assume that each element of A and B is jointly independently and identically distributed through time, so the expected value operations need not be time-conditional.

Induction backwards in time can be used to obtain the optimal control solution at each time,[2]^{: ch. 13}

${\displaystyle u_{t}^{*}=-\left[\mathrm {E} \left(B^{\mathsf {T}}X_{t}B+R\right)\right]^{-1}\mathrm {E} \left(B^{\mathsf {T}}X_{t}A\right)y_{t-1},}$

with the symmetric positive definite cost-to-go matrix X evolving backwards in time from ${\displaystyle X_{S}=Q}$ according to

${\displaystyle X_{t-1}=Q+\mathrm {E} \left[A^{\mathsf {T}}X_{t}A\right]-\mathrm {E} \left[A^{\mathsf {T}}X_{t}B\right]\left[\mathrm {E} (B^{\mathsf {T}}X_{t}B+R)\right]^{-1}\mathrm {E} \left(B^{\mathsf {T}}X_{t}A\right),}$

which is known as the discrete-time dynamic Riccati equation of this problem. The only information needed regarding the unknown parameters in the A and B matrices is the expected value and variance of each element of each matrix and the covariances among elements of the same matrix and among elements across matrices.

The optimal control solution is unaffected if zero-mean, i.i.d. additive shocks also appear in the state equation, so long as they are uncorrelated with the parameters in the A and B matrices. But if they are so correlated, then the optimal control solution for each period contains an additional additive constant vector. If an additive constant vector appears in the state equation, then again the optimal control solution for each period contains an additional additive constant vector.

The steady-state characterization of X (if it exists), relevant for the infinite-horizon problem in which S goes to infinity, can be found by iterating the dynamic equation for X repeatedly until it converges; then X is characterized by removing the time subscripts from its dynamic equation.

In a continuous time approach in a finance context, the state variable in the stochastic differential equation is usually wealth or net worth, and the controls are the shares placed at each time in the various assets. Given the asset allocation chosen at any time, the determinants of the change in wealth are usually the stochastic returns to assets and the interest rate on the risk-free asset. The field of stochastic control has developed greatly since the 1970s, particularly in its applications to finance. Robert Merton used stochastic control to study optimal portfolios of safe and risky assets.[7] His work and that of Black–Scholes changed the nature of the finance literature. Influential mathematical textbook treatments were by Fleming and Rishel,[8] and by Fleming and Soner.[9] These techniques were applied by Stein to the financial crisis of 2007–08.[10]

The maximization, say of the expected logarithm of net worth at a terminal date T, is subject to stochastic processes on the components of wealth.[11] In this case, in continuous time Itô's equation is the main tool of analysis. In the case where the maximization is an integral of a concave function of utility over an horizon (0,T), dynamic programming is used. There is no certainty equivalence as in the older literature, because the coefficients of the control variables—that is, the returns received by the chosen shares of assets—are stochastic.