Asymptotic Analysis of a Deterministic Control System via Euler ’ s Equation Approach

This paper provides necessary conditions in order to guarantee the existence of an unique equilibrium point in a deterministic control system. Furthermore, under additional conditions, it is proved the convergence of the optimal control sequence to this equilibrium point. The methodology to obtain these statements is based on the Euler’s equation approach. A consumption-investment problem is presented with the objective to illustrate the results exposed.


Introduction
In this document a Deterministic Control System (DCS) is considered.A DCS is used to modelling dynamic systems, which are observed in a discrete time by a controller with the purpose that the system performs effectively with respect to certain optimality criteria (objective function).In this way a sequence of strategies to operate the dynamic system is obtained, this sequence is denominated a control policy.The optimal control problems consist in determining an optimal policy, i.e. a policy that minimizes the objective function.These classes of problems are included in the theory of Markov Decision Processes (Hernández-Lerma & Jean, 1996;Hinderer, Rieder & Stieglitz, 2017).In this context, a problem of interest is analysing the asymptotic behaviour of the optimal trajectory of the system, this problem is addressed in this document.
The convergence of the optimal trajectory of the DCS becomes of interest, in this work the necessary conditions for the convergence to its respective equilibrium point are provided.Important investigations base their results on the existence and characterization of the equilibrium point of the deterministic system associated with the stochastic system being studied (Judd & Guu, 1993).The work is motivated from studying the economic growth model as it is exposed in De la Fuente (2000), where the stability of the system is guaranteed.In this document, we propose conditions for the stability of more general systems.The methodology that follows is to prove that the optimal trajectory is monotonous and bounded, thus demonstrating that it is convergent, later, the equilibrium point is characterized by the Euler's equation.Finally, it is proved that the convergence to the equilibrium point holds.This paper is organized as follows.In Section 2, the deterministic system of control, main concepts and results in the theory of DCS are presented.In section 3 an essential result for the main objective of this work is developed.Finally, in Section 4, some concluding remarks are provided.
Notation: In the following sections, the derivative of a multi-variable function M with respect to a variable y, it will be denoted by M y , the gradient of the function M is denoted by DM(x, y) and the derivative of a function of a variable h will be denoted by h ′ .

Materials and Methods
A deterministic control model is conformed by the following components: where X = [0, ∞), is called the state space, A = [0, ∞) is denominated the action space.{A(x)|x ∈ X} is a family of measurable non-empty subsets A(x) of A, here A(x) denotes the set of feasible actions when the system is in state The interpretation of the model is as follows: at each time t ∈ {0, 1, 2, ...}, we observe the system state x t = x ∈ X, we apply an action a t = a ∈ A(x), and as a result we receive a reward r(x, a) and the system is transferred to a new state x t+1 , determined by x t+1 = F(x t , a t ), t = 0, 1, ..., with x 0 = x ∈ X initial state.
A deterministic policy is a sequence π = { f t : t = 0, 1, ...} such that f t ∈ F, t = 0, 1, ....The set of deterministic Markovian policies is denoted by M. In particular, if f t = f ∈ F, for all t ∈ {0, 1, 2, ...}, π = { f, f, ...} is called a stationary policy, and in this case the policy π is only denoted by f .Define for each x ∈ X and π ∈ M ) is called the total discounted reward, where α ∈ (0, 1) is the discount factor.
The objective of the control problem is to maximize π → v(π, x) on M, for all x ∈ X, i.e., determining the optimal policy π * and optimal value function V, which are defined as follows.
V is called the optimal value function or optimal value.
The following assumptions will be applied in the rest of the paper.
b) r and F are continuous functions on K.
c) There exist constants c and β, with 1 ≤ β < 1/α, and a function w Assumptions II a) There exists a function H : X → X such that sup a∈A(X) F(x, a) ≤ H(x), for each x ∈ X.Furthermore, for the function H, there exists Remark 1 In order to guarantee the differentiability of the optimal solution (see Assumption II d) and e)), you can consult the following references: De la Fuente (2000), Hernández-Lerma & Jean (1996) and Cruz & Montes- de-Oca (2008).
Similarly, the concavity of the optimal value function and the monotonicity of the optimal policy can be consulted in Cruz, Montes-de-Oca & Salem (2004) and Flores (2013), respectively.
The following theorem characterizes the optimal solution of the optimal control problem, via dynamic programming approach.
Theorem 1 Suppose that Assumptions I hold, then: a) The optimal value function V is a solution for the following equation known as the Dynamic Programming Equation: x ∈ X and f is optimal.c) V n (x) → V(x), as n → ∞, for each x ∈ X (see ( 1)).
Then, for Theorem 1 b), there exists f ∈ F such that: t = 1, 2, ..., where {x * t } represents the optimal trajectory of the control process.In this paper we are interested in studying the behaviour of (4) at its equilibrium point.According to Judd (1998), an equilibrium point x ∈ X associated with (4), satisfies the following identity: x = F( x, f ( x)) = g( x). (5)

Main Results
This section presents the main result, which is to demonstrate that there exists an equilibrium point of ( 4) and the convergence of the optimal trajectory to this point.
Theorem 2 If r and F are concave functions and r, F ∈ C 2 (K) then the optimal policy, f , satisfies the following Euler equation: where ∆(x, a) := (r x − r a F x /F a )(x, a), (x, a) ∈ K.
Conversely, if f ∈ F is a policy which satisfies (6) for each x ∈ (0, ∞) and then f is an optimal policy.
Proof.Let f be the optimal policy.Then by (2): {r(x, a) + αV(F(x, a))} , in consequence the first-order condition is given by On the other hand, by (3): substituting ( 8) in (9), it is obtained that In consequence, from ( 8) and ( 10) follows that: Finally, substituting (11) in ( 8), the results follows.Now, will prove the converse.Let f be a function that satisfies ( 6) and ( 7) and consider x ∈ (0, ∞) fixed.Let f ∈ F be another function and for t = 0, 1, ..., the trajectories of the policies f and f be denoted by x t and xt , respectively.Similarly, a t = f (x t ) and ât = f ( xt ) denote their respective actions, where x 0 = x0 = x for both.
Since r is concave and r ∈ C 2 , applying Theorem 2.17 in De La Fuente (2000), p.258, it is obtained that for T > 1, T is a positive integer.Since x t+1 = F(x t , a t ) and xt+1 = F( xt , ât ), then Since F is concave and F ∈ C 2 , it results that in consequence, it is obtained that Now, since a t = f (x t ) and f satisfies (6), it follows that Now by ( 12), it is possible to conclude that Similarly, Then letting T → ∞, by ( 7), it follows that Therefore f is an optimal policy.
Theorem 3 Suppose that r and F are strictly concave functions and the Assumptions I and II hold.Then there exists an unique equilibrium point x ∈ X of g.
Proof.Firstly, observe that the equilibrium point x, satisfies the following system equations: • if x 0 > x then since g is increasing (Remark 2.a), x * t > x.In consequence, W(x * t ) < W( x) = 0 and (17) implies that x * t < x * t+1 , that is {x * t } is a monotonous increasing sequence.
• Analogously, if x 0 < x then {x * t } is a monotonous decreasing sequence.
Remark 3 Note that if x 0 < x, by the proof of Proposition 1, {x * t } increases and if the Assumption II a) holds, then x < x M , in consequence x 0 ≤ x * t ≤ x M .Otherwise, x 0 > x, then 0 ≤ x * t ≤ x 0 .In this way, the compact in which f n converge to f can be defined as S := [0, max{x M , x 0 }].
Theorem 4 The optimal trajectory {x * t } converge monotonously into S ⊆ X to equilibrium point x for any initial value x 0 ∈ X.
Proof.Since {x * t } is a monotonous and bounded sequence (see Proposition 1 and Remark 3, respectively), then it converges.Let x * be the limit of sequence {x * t }.The, for the continuity of the function g (Remark 2.b), x * must be a fixed point, because Then x * is an equilibrium point, but as x is the only equilibrium point, x * t → x.

Example: Logarithmic Utility
Consider an economy in which at each discrete time t, t = 0, 1, ..., there are L t consumers (population or labor), with consumption a t per individual, whose growth is governed by the following difference equation: it is assumed that initially the number of consumers, L 0 , is known.In this case η > 0 fixed represents an exogenous shock that affects the consumer population.In general, this shock is considered as a random variable, in this context η represents the expected value (see Vitoriano, Werra, & Parlier, 2016).The production function for the economy is given by with K 0 known, i.e. the production Y t is a function of capital, K t , and labor, L t , where the production function, G, is a homogeneous function of degree one, i.e.G(λx, λy) = λG(x, y).The output must be split between consumptions C t = a t L t and the gross investment I t , i.e.
Let δ ∈ (0, 1) be the depreciation rate of capital.Then the evolution equation for capital is given by: Substituting ( 20) in ( 19), it is obtained that In the usual way, all variables can be normalized into per capital terms, namely, y t := Y t /L t and x t := K t /L t .Then ( 21) can be expressed in the following way: Now, using (18) in the previous relation, it yields that ), h henceforth to be identified as the production function.
Lemma 1 The logarithmic utility example satisfies Assumption I.
Proof.Firstly, observe that r(x, a) = ln(a) if a ∈ (0, x γ ], and r( due to r and F ∈ C 2 (see Remark 1 and Cruz-Suárez ( 2008)).Furthermore, the second derivative of the value function si strictly positive, since r and F are strictly concave and increasing functions and the multifunction, x → A(x), is increasing and convex function.Proof.The equilibrium point is characterized by equations ( 5) and ( 15).Then substituting the respective values in ( 5) and ( 15), it results that x. Solving the previous system for x and f ( x), it is obtained that: x = ( η αγ In conclusion, for this example, the optimal trajectory converges to x = ( η αγ The Figure 1 represents the first ten realization of the optimal trajectory.

Conclusions
In this work conditions on the model which guarantee the convergence of the optimal trajectory of the DCS are established.
In specific, a compact subset S of the real numbers is characterized (see Remark 3).Then, it is proved that the optimal trajectory is concentrated in the set S. Finally, it is shown that the optimal trajectory is converging to the equilibrium point of the DCS.Future works in this direction, consists in studying a stochastic control problem induced by the DCS (Cruz & Montes- de-Oca, 2006), and in this case, we are interested in guaranteeing the convergence of the stochastic optimal trajectory.
, (x, a) ∈ K.In consequence, Assumption I a) and Assumption I b) hold.Then it just necessary to prove Assumption I c), for this case consider w(x) = ln(x/η), x ∈ [0, ∞) and c = 1 = β.Lemma 2 The logarithmic utility example satisfies Assumptions II.Proof.a) Let H(x) = x γ /η, be a continuous function mapping defined on the compact and convex set X = [0, ∞).Then by Theorem 3.2 in De la Fuente (2000) p. 221, H has a fixed point in X, this is indeed x e) Since r and F ∈ C 2 , by Remark 1, f ∈ C 1 and due to the multifunction x → A(x) is increasing and convex, by Remark 3, f is increasing.Lemma 3 The equilibrium point is x = ( Figure 1.Optimal Trajectory