Contribution to the Study of Backward SDEs and Their Applications to Stochastic Optimal Control

In this thesis, we aim to generalize some existing results in the literature that concern a stochastic maximum principle for backward stochastic differential equations (BSDEs) or forward-backward stochastic differential equation (FBSDEs), with two possible directions. The first direction is concerned with the stochastic control problem for BSDEs with locally Lipschitz generators, where the domain is not necessarily convex, we establish a necessary and sufficient condition for optimality satisfied by all optimal controls. These conditions are described by a linear locally Lipschitz SDE and a maximum condition on the Hamiltonian. We first prove, under some convenient conditions, the existence of a unique solution to the resulting adjoint equation. Then, with the help of an approximation argument on the coefficients, we define a family of control problems with globally Lipschitz coefficients whereby we derive a stochastic maximum principle for near optimality to such approximated systems. Thereafter, we turn back to the original control problem by passing to the limits. The second direction is devoted to the stochastic maximum principle in optimal control of possibly degenerate FBSDEs, with irregular coefficients. We assume that the coefficients satisfy the Lipschitz conditions, the control domain is non-convex and the control variable does not enter to the diffusion coefficient. We obtain the necessary conditions for optimality utilizing an adjoint process, which is the unique solution of a linear backward-forward stochastic differential equation and a maximal condition on the Hamiltonian. Thanks to the Bouleau-Hirsch flow property, we are able to define the adjoint process employing the derivatives of the coefficients in the sense of distributions. Moreover, the adjoint process does not depend on the choice of the representatives of the weak derivatives.