Hamilton-Jacobi-Bellman equation

The Hamilton-Jacobi-Bellman (HJB) equation is a partial differential equation which is central to optimal control theory.

Related Topics:
Partial differential equation - Optimal control

~ ~ ~ ~ ~ ~ ~ ~ ~ ~

The solution of the HJB equation is the 'value function', which gives the optimal cost-to-go for a given dynamical system with an associated cost function. Classical variational problems, for example, the brachistochrone problem can be solved using this method as well.

Related Topics:
Dynamical system - Brachistochrone problem

~ ~ ~ ~ ~ ~ ~ ~ ~ ~

The equation is a result of the theory of dynamic programming which was pioneered in the 1950s by Richard Bellman and coworkers. The corresponding discrete-time equation is usually referred to as the Bellman equation. In continuous time, the result can be seen as an extension of earlier work in classical physics by William Rowan Hamilton and Carl Gustav Jacob Jacobi.

Related Topics:
Dynamic programming - Richard Bellman - Bellman equation - Classical physics - William Rowan Hamilton - Carl Gustav Jacob Jacobi

~ ~ ~ ~ ~ ~ ~ ~ ~ ~

Consider the following problem in deterministic optimal control

~ ~ ~ ~ ~ ~ ~ ~ ~ ~

: min int_0^T C + D,dt

~ ~ ~ ~ ~ ~ ~ ~ ~ ~

subject to

~ ~ ~ ~ ~ ~ ~ ~ ~ ~

: dot{x}(t)=F

~ ~ ~ ~ ~ ~ ~ ~ ~ ~

where x(t) is the system state, x(0) is assumed given, and u(t) for 0leq tleq T is the control that we are trying to find.

~ ~ ~ ~ ~ ~ ~ ~ ~ ~

For this simple system, the Hamilton Jacobi Bellman partial differential equation is

~ ~ ~ ~ ~ ~ ~ ~ ~ ~

:

~ ~ ~ ~ ~ ~ ~ ~ ~ ~

0 = C(x,u) + rac{partial}{partial t} V(x,t) + left^prime F(x,u)

~ ~ ~ ~ ~ ~ ~ ~ ~ ~

~ ~ ~ ~ ~ ~ ~ ~ ~ ~

subject to the terminal condition

~ ~ ~ ~ ~ ~ ~ ~ ~ ~

:

~ ~ ~ ~ ~ ~ ~ ~ ~ ~

V(x,T) = D(x).,

~ ~ ~ ~ ~ ~ ~ ~ ~ ~

~ ~ ~ ~ ~ ~ ~ ~ ~ ~

The unknown V(x, t) in the above PDE is the Bellman 'value function', that is the cost incurred from starting in state x at time t and controlling the system optimally from then until time T.

~ ~ ~ ~ ~ ~ ~ ~ ~ ~

The HJB equation needs to be solved backwards in time, starting from t = T and ending at t = 0.

~ ~ ~ ~ ~ ~ ~ ~ ~ ~

The HJB equation is a sufficient condition for an optimum. If we can solve for V then we can find from it a control u that achieves the minimum cost.

~ ~ ~ ~ ~ ~ ~ ~ ~ ~

The HJB method can be generalized to stochastic systems as well.

~ ~ ~ ~ ~ ~ ~ ~ ~ ~