Separation principle in stochastic control

Linear-quadratic control problems are often solved by a completion-of-squares argument. In our present context we have

${\displaystyle J(u)=\operatorname {E} \left\{\int _{0}^{T}(u-Kx)'R(u-Kx)\,dt\right\}+{\text{terms that do not depend on }}u,}$

in which the first term takes the form[3]

${\displaystyle {\begin{aligned}\operatorname {E} \left\{\int _{0}^{T}(u-Kx)'R(u-Kx)\,dt\right\}=\operatorname {E} \left\{\int _{0}^{T}[(u-K{\hat {x}})'R(u-K{\hat {x}})+\operatorname {tr} (K'RK\Sigma )]\,dt\right\},\end{aligned}}}$

where ${\displaystyle \Sigma }$ is the covariance matrix

${\displaystyle \Sigma (t):=\operatorname {E} \{[x(t)-{\hat {x}}(t)][x(t)-{\hat {x}}(t)]'\}.}$

The separation principle would now follow immediately if ${\displaystyle {\begin{aligned}\Sigma \end{aligned}}}$ were independent of the control. However this needs to be established.

The state equation can be integrated to take the form

${\displaystyle x(t)=x_{0}(t)+\int _{0}^{t}\Phi (t,s)B_{1}(s)u(s)\,ds,}$

where ${\displaystyle x_{0}}$ is the state process obtained by setting ${\displaystyle u=0}$ and ${\displaystyle \Phi }$ is the transition matrix function. By linearity, ${\displaystyle {\hat {x}}(t)=\operatorname {E} \{x(t)\mid {\cal {Y}}_{t}\}}$ equals

${\displaystyle {\hat {x}}(t)={\hat {x}}_{0}(t)+\int _{0}^{t}\Phi (t,s)B_{1}(s)u(s)\,ds,}$

where ${\displaystyle {\hat {x}}_{0}(t)=\operatorname {E} \{x_{0}(t)\mid {\cal {Y}}_{t}\}}$. Consequently,

${\displaystyle \Sigma (t):=\mathbb {E} \{[x_{0}(t)-{\hat {x}}_{0}(t)][x_{0}(t)-{\hat {x}}_{0}(t)]'\},}$

but we need to establish that ${\displaystyle {\begin{aligned}{\hat {x}}_{0}\end{aligned}}}$ does not depend on the control. This would be the case if

${\displaystyle {\cal {Y}}_{t}={\cal {Y}}_{t}^{0}:=\sigma \{y_{0}(\tau ),\tau \in [0,t]\},\quad 0\leq t\leq T,}$

where ${\displaystyle y_{0}}$ is the output process obtained by setting ${\displaystyle u=0}$. This issue was discussed in detail by Lindquist.[2] In fact, since the control process ${\displaystyle u}$ is in general a nonlinear function of the data and thus non-Gaussian, then so is the output process ${\displaystyle y}$. To avoid these problems one might begin by uncoupling the feedback loop and determine an optimal control process in the class of stochastic processes ${\displaystyle u}$ that are adapted to the family ${\displaystyle \{{\cal {Y}}_{t}^{0}\}}$ of sigma fields. This problem, where one optimizes over the class of all control processes adapted to a fixed filtration, is called a stochastic open loop (SOL) problem.[2] It is not uncommon in the literature to assume from the outset that the control is adapted to ${\displaystyle \{{\mathcal {Y}}_{t}^{0}\}}$; see, e.g., Section 2.3 in Bensoussan,[4] also van Handel [5] and Willems.[6]

In Lindquist 1973[2] a procedure was proposed for how to embed the class of admissible controls in various SOL classes in a problem-dependent manner, and then construct the corresponding feedback law. The largest class ${\displaystyle \Pi }$ of admissible feedback laws ${\displaystyle \pi }$ consists of the non-anticipatory functions ${\displaystyle u:=\pi (y)}$ such that the feedback equation has a unique solution and the corresponding control process ${\displaystyle u_{\pi }}$ is adapted to ${\displaystyle \{{\mathcal {Y}}_{t}^{0}\}}$. Next, we give a few examples of specific classes of feedback laws that belong to this general class, as well as some other strategies in the literature to overcome the problems described above.

At this point it is suitable to consider a more general class of controlled linear stochastic systems that also covers systems with time delays, namely

${\displaystyle {\begin{aligned}z(t)&=z_{0}(t)+\int _{0}^{t}G(t,s)u(s)\,ds\\y(t)&=Hz(t)\end{aligned}}}$

with ${\displaystyle {\begin{aligned}z_{0}\end{aligned}}}$ a stochastic vector process which does not depend on the control.[2] The standard stochastic system is then obtained as a special case where ${\displaystyle z=[x',y']'}$, ${\displaystyle z_{0}=[x_{0}',y_{0}']'}$ and ${\displaystyle H=[I,0]}$. We shall use the short-hand notation

${\displaystyle z=z_{0}+g\pi Hz}$

for the feedback system, where

${\displaystyle g\;:\;(t,u)\mapsto \int _{0}^{t}G(t,\tau )u(\tau )\,d\tau }$

is a Volterra operator.

In this more general formulation the embedding procedure of Lindquist[2] defines the class ${\displaystyle \Pi }$ of admissible feedback laws ${\displaystyle \pi }$ as the class of non-anticipatory functions ${\displaystyle u:=\pi (y)}$ such that the feedback equation ${\displaystyle z=z_{0}+g\pi Hz}$ has a unique solution ${\displaystyle z_{\pi }}$ and ${\displaystyle u=\pi (Hz_{\pi })}$ is adapted to ${\displaystyle \{{\mathcal {Y}}_{t}^{0}\}}$.

In Georgiou and Lindquist[1] a new framework for the separation principle was proposed. This approach considers stochastic systems as well-defined maps between sample paths rather than between stochastic processes and allows us to extend the separation principle to systems driven by martingales with possible jumps. The approach is motivated by engineering thinking where systems and feedback loops process signals, and not stochastic processes per se or transformations of probability measures. Hence the purpose is to create a natural class of admissible control laws that make engineering sense, including those that are nonlinear and discontinuous.

The feedback equation ${\displaystyle z=z_{0}+g\pi Hz}$ has a unique strong solution if there exists a non-anticipating function ${\displaystyle F}$ such that ${\displaystyle z=F(z_{0})}$ satisfies the equation with probability one and all other solutions coincide with ${\displaystyle z}$ with probability one. However, in the sample-wise setting, more is required, namely that such a unique solution exists and that ${\displaystyle z=z_{0}+g\pi Hz}$ holds for all ${\displaystyle z_{0}}$, not just almost all. The resulting feedback loop is deterministically well-posedin the sense that the feedback equations admit a unique solution that causally depends on the input for each input sample path.

In this context, a signal is defined to be a sample path of a stochastic process with possible discontinuities. More precisely, signals will belong to the Skorohod space ${\displaystyle D}$, i.e., the space of functions which are continuous on the right and have a left limit at all points (càdlàg functions). In particular, the space ${\displaystyle C}$ of continuous functions is a proper subspace of ${\displaystyle D}$. Hence the response of a typical nonlinear operation that involves thresholding and switching can be modeled as a signal. The same goes for sample paths of counting processes and other martingales. A system is defined to be a measurable non-anticipatory map ${\displaystyle D\to D}$ sending sample paths to sample paths so that their outputs at any time ${\displaystyle t}$ is a measurable function of past values of the input and time. For example, stochastic differential equations with Lipschitz coefficients driven by a Wiener process induce maps between corresponding path spaces, see page 127 in Rogers and Williams,[16] and pages 126-128 in Klebaner.[17] Also, under fairly general conditions (see e.g., Chapter V in Protter[18]), stochastic differential equations driven by martingales with sample paths in ${\displaystyle D}$ have strong solutions who are semi-martingales.

For the time setting ${\displaystyle f(z):=g\pi Hz}$, the feedback system ${\displaystyle z=z_{0}+g\pi Hz}$ can be written ${\displaystyle z=z_{0}+f(z)}$, where ${\displaystyle z_{0}}$ can be interpreted as an input.

Definition. A feedback loop ${\displaystyle z=z_{0}+f(z)}$ is deterministically well-posed if it has a unique solution ${\displaystyle z\in D}$ for all inputs ${\displaystyle z_{0}\in D}$ and ${\displaystyle (1-f)^{-1}}$ is a system.

This implies that the processes ${\displaystyle z}$ and ${\displaystyle z_{0}}$ define identical filtrations.[1] Consequently, no new information is created by the loop. However, what we need is that ${\displaystyle {\cal {Y}}_{t}={\cal {Y}}_{t}^{0}}$ for ${\displaystyle 0\leq t\leq T}$. This is ensured by the following lemma (Lemma 8 in Georgiou and Lindquist[1]).

Key Lemma. If the feedback loop ${\displaystyle z=z_{0}+g\pi Hz}$ is deterministically well-posed, ${\displaystyle g\pi }$ is a system, and ${\displaystyle H}$ is a linear system having a right inverse ${\displaystyle H^{-R}}$ that is also a system, then ${\displaystyle (1-Hg\pi )^{-1}}$ is a system and ${\displaystyle {\cal {Y}}_{t}={\cal {Y}}_{t}^{0}}$ for ${\displaystyle 0\leq t\leq T}$.

The condition on ${\displaystyle H}$ in this lemma is clearly satisfied in the standard linear stochastic system, for which ${\displaystyle H=[0,I]}$, and hence ${\displaystyle H^{-R}=H'}$. The remaining conditions are collected in the following definition.

Definition. A feedback law ${\displaystyle \pi }$ is deterministically well-posed for the system ${\displaystyle z=z_{0}+g\pi Hz}$ if ${\displaystyle g\pi }$ is a system and the feedback system ${\displaystyle z=z_{0}+g\pi Hz}$ deterministically well-posed.

Examples of simple systems that are not deterministically well-posed are given in Remark 12 in Georgiou and Lindquist.[1]

By only considering feedback laws that are deterministically well-posed, all admissible control laws are physically realizable in the engineering sense that they induce a signal that travels through the feedback loop. The proof of the following theorem can be found in Georgiou and Lindquist 2013.[1]

Separation theorem. Given the linear stochastic system

${\displaystyle {\begin{aligned}dx&=A(t)x(t)\,dt+B_{1}(t)u(t)\,dt+B_{2}(t)\,dw\\dy&=C(t)x(t)\,dt+D(t)\,dw\end{aligned}}}$

where ${\displaystyle w}$ is a vector-valued Wiener process, ${\displaystyle x(0)}$ is a zero-mean Gaussian random vector independent of ${\displaystyle w}$, consider the problem of minimizing the quadratic functional J(u) over the class of all deterministically well-posed feedback laws ${\displaystyle \pi }$. Then the unique optimal control law is given by ${\displaystyle u(t)=K(t){\hat {x}}(t)}$ where ${\displaystyle K}$ is defined as above and ${\displaystyle {\hat {x}}}$ is given by the Kalman filter. More generally, if ${\displaystyle w}$ is a square-integrable martingale and ${\displaystyle x(0)}$ is an arbitrary zero mean random vector, ${\displaystyle u(t)=K(t){\hat {x}}(t)}$, where ${\displaystyle {\hat {x}}(t)=\operatorname {E} \{x(t)\mid {\cal {Y}}_{t}\}}$, is the optimal control law provided it is deterministically well-posed.

In the general non-Gaussian case, which may involve counting processes, the Kalman filter needs to be replaced by a nonlinear filter.

Stochastic control for time-delay systems were first studied in Lindquist,[19][20][8][2] and Brooks,[21] although Brooks relies on the strong assumption that the observation ${\displaystyle y}$ is functionally independent of the control ${\displaystyle u}$, thus avoiding the key question of feedback.

Consider the delay-differential system[8]

${\displaystyle {\begin{aligned}dx&=\left(\int _{t-h}^{t}d_{s}\,A(t,s)x(s)\right)\,dt+B_{1}(t)u(t)\,dt+B_{2}(t)\,dw\\dy&=\left(\int _{t-h}^{t}d_{s}\,C(t,s)x(s)\right)\,dt+D(t)\,dw\end{aligned}}}$

where ${\displaystyle w}$ is now a (square-integrable) Gaussian (vector) martingale, and where ${\displaystyle {\begin{aligned}A\end{aligned}}}$ and ${\displaystyle C}$ are of bounded variation in the first argument and continuous on the right in the second, ${\displaystyle x(t)=\xi (t)}$ is deterministic for ${\displaystyle -h\leq t\leq 0}$, and ${\displaystyle y(0)=0}$. More precisely, ${\displaystyle A(t,s)=0}$ for ${\displaystyle s\geq t}$, ${\displaystyle A(t,s)=A(t,t-h)}$ for ${\displaystyle t\leq t-h}$, and the total variation of ${\displaystyle s\mapsto A(t,s)}$ is bounded by an integrable function in the variable ${\displaystyle t}$, and the same holds for ${\displaystyle C}$.

We want to determine a control law which minimizes

${\displaystyle J(u)=\operatorname {E} \left(\int _{0}^{T}x(t)'Q(t)x(t)\,d\alpha (t)+\int _{0}^{T}u(t)'R(t)u(t)\,dt\right),}$

where ${\displaystyle {\begin{aligned}d\alpha \end{aligned}}}$ is a positive Stieltjes measure. The corresponding deterministic problem obtained by setting ${\displaystyle {\begin{aligned}w=0\end{aligned}}}$ is given by

${\displaystyle u(t)=\int _{t-h}^{t}d_{\tau }\,K(t,\tau )x(\tau ),}$

with[8] ${\displaystyle {\begin{aligned}K\end{aligned}}}$.

The following separation principle for the delay system above can be found in Georgiou and Lindquist 2013[1] and generalizes the corresponding result in Lindquist 1973[8]

Theorem. There is a unique feedback law ${\displaystyle {\begin{aligned}\pi$ :\,y\mapsto u\end{aligned}}} in the class of deterministically well-posed control laws that minimizes ${\displaystyle {\begin{aligned}J(u)\end{aligned}}}$, and it is given by

${\displaystyle u(t)=\int _{t-h}^{t}d_{s}\,K(t,s){\hat {x}}(s\mid t),}$

where ${\displaystyle K}$ is the deterministic control gain and ${\displaystyle {\hat {x}}(s\mid t):=E\{x(s)\mid {\cal {Y}}_{t}\}}$ is given by the linear (distributed) filter

${\displaystyle {\begin{aligned}d{\hat {x}}(t\mid t)&=\int _{t-h}^{t}d_{s}\,A(t,s){\hat {x}}(s\mid t)\,dt+B_{1}u\,dt+X(t,t)\,dv\\d{\hat {x}}(t\mid t)&=\int _{t-h}^{t}d_{s}\,A(t,s){\hat {x}}(s\mid t)\,dt+B_{1}u\,dt+X(t,t)\,dv\end{aligned}}}$

where ${\displaystyle v}$ is the innovation process

${\displaystyle dv=dy-\int _{t-h}^{t}d_{s}C(t,s){\hat {x}}(s\mid t)\,dt,\quad v(0)=0,}$

and the gain ${\displaystyle x}$ is as defined in page 120 in Lindquist.[8]