Markov decision processes approximation with coupled dynamics via Markov deterministic control systems

1 Introduction

This article deals with the so-called discrete-time Markov decision processes (MDPs) with an infinite horizon and total discounted cost [1–5]. The importance of working with MDPs lies in the wide range of applications in various disciplines, e.g., engineering, computer science, communications, and economics [6,7]. The main problem in MDPs is to determine an optimal policy and the optimal value function. To characterize and determine the solutions of MDPs, the dynamic programming (DP) approach [2,8] is available.

In this work, the MDPs of interest are those that evolve through dynamics consisting of two coupled difference equations, as shown in equations (1) and (2). Equation (1) models the transitions of system states, where the set of all states is denoted by X and its elements are called x -states. Similarly, equation (2) models the change of the system’s parameters; the set of all parameters is denoted by Γ , and its elements are called α -states. Let ε 0 and δ 0 be positive numbers and let ε ∈ [ 0 , ε 0 ] and δ ∈ [ 0 , δ 0 ] , then we consider disturbances { ξ t ( ε ) } and { η t ( δ ) } , which are sequences of independent and identically distributed random elements with values in some Borel spaces ( S 1 , r 1 ) and ( S 2 , r 2 ) (metric perturbation spaces [9] or noise spaces [10]), respectively. Moreover, suppose that there exist s 1 ∈ S 1 and s 2 ∈ S 2 such that s 1 = ξ t ( 0 ) and s 2 = η t ( 0 ) , each element of the above sequences depends on numerical parameters ε and δ such that E r 1 ( ξ ( ε ) , s 1 ) → 0 when ε → 0 and E r 2 ( η ( δ ) , s 2 ) → 0 when δ → 0 , where ξ and η are generic elements of { ξ t ( ε ) } and { η t ( δ ) } , respectively. In this framework, we are interested in the following problems:

• To study approximations of MDPs by the deterministic control process (see equations (3) and (4)). In particular, we are interested in ensuring that the optimal policy of the deterministic system is asymptotically optimal for the random system (see Theorem 1 and Remark 4).

• To analyze the convergence of the optimal value function and the optimal policy of the stochastic system when ε → 0 and δ → 0 (see Theorem 2).

The following briefly describes work related to the problems discussed in this manuscript. In a study by Liptser et al. [11], the problem of approximating a continuous-time stochastic control process by a deterministic process was considered. In this article, the authors demonstrate that the stochastic problem can be approximated by a deterministic one when the noise is small and the fluctuations become fast. In this context, it is shown that the optimal control of the deterministic problem is asymptotically optimal for stochastic problems. In the continuous case, Dupuis and Kushner [12] addressed a similar problem, i.e., when the effects of noise in a physical system are small, these authors performed an asymptotic analysis of the diffusion approximation and used it for the desired estimates in the original system. For discrete-time MDPs, these classes of problems were studied by Cruz-Suarez and Ilhuicatzi-Roldan [9] and Cruz-Suarez et al. [13], where the dynamic of the system is described by a single difference equation. Convergence between models was also addressed in by Kara and Yuksel [14]. However, convergence is studied using sequences belonging to the set of admissible state-action pairs, which is assumed to be a subset of a given Euclidean space. Moreover, this study is carried out under the assumption that the action space is a compact set and that the cost function is bounded. Now, when considering MDPs that are developed with respect to equations (1) and (2), the results found in the study by Cruz-Suarez et al. [13] are generalized. The approach of using coupled equations can be applied, e.g., by considering a random discount factor [15–18], where the second difference equation refers to the evolution of the random discount factor.

The methodology for solving the problems described above is to impose Lipschitz continuity [19,20] constraints on the components of the control model and to apply DP techniques. Specifically, we assume Lipschitz conditions for the functions c , F , and G involved in the dynamic system composed of two coupled difference equations (see equations (1) and (2)). A direct consequence of this assumption is the Lipschitz continuity of the optimal cost, which corresponds to an additional contribution to the present manuscript. This approach ensures the following three important aspects:

• The existence of an upper bound for the stability index [10,21,22] when we apply the optimal policy of the deterministic system. Consequently, it results that the optimal policy of the deterministic system is asymptotically optimal for the stochastic system (see Theorem 1 and Remark 4).

• A convergence rate of the optimal cost function for the random system with respect to the deterministic system.

• The uniform convergence of the optimal stochastic policy to the deterministic policy when ε → 0 and δ → 0 on compact subsets of the state space.

This article is structured as follows: Section 2 presents the basic theory of MDPs with states evolving with dynamics consisting of two coupled difference equations; Section 3 provides the approximation problem statement for the value function and the optimal policy; Section 4 presents the results that provide the bound for the stability index δ ˆ ε , δ , the convergence rate of the optimal cost, and the convergence of the optimal policy on compact subsets; Section 5 illustrates the developed theory with two examples. The first relates to a consumption-investment problem [15,23]. The second example is a control problem with small additive noise. For both problems, the upper bounds for the stability index and the convergence rate of the optimal value function are given explicitly; and finally, in Section 6, concluding remarks are given.

2 Markov control model

Consider the following Markov model:

where X × Γ and A are Borel spaces and are called the state space and the action space, respectively; { A ( x , α ) ∣ ( x , α ) ∈ X × Γ } is a family of non-empty measurable subsets A ( x , α ) of A , where A ( x , α ) denotes the set of feasible actions (controls) when the system is in state ( x , α ) ∈ X × Γ . The set K of feasible state-actions is defined as follows:

which is a measurable subset of X × Γ × A ; the next component is a stochastic kernel Q on X × Γ given K , i.e., Q ( ⋅ ∣ x , α , a ) is a probability measure on X × Γ for each ( x , α , a ) ∈ K and Q ( B ∣ ⋅ ) is a measurable function on K for each B ∈ ℬ ( X × Γ ) , where ℬ ( X × Γ ) denotes the Borel σ -algebra of X × Γ ; c : K → R is a measurable function called the one-stage cost function.

The dynamic of the system is described below. Suppose that at time t , t ∈ { 0 , 1 , … } , the system occupies state ( x t , α t ) = ( x , α ) ∈ X × Γ . Then, the decision-maker (or controller) chooses a control a t = a ∈ A ( x , α ) . Consequently, two things happen:

1. a cost c ( x t , α t , a t ) is incurred, and

2. the system jumps to a state ( x t + 1 , α t + 1 ) = ( x ′ , α ′ ) according to the transition law Q ( ⋅ ∣ x , α , a ) (i.e., Q ( B ∣ x , α , a ) = P r ( ( x t + 1 , α t + 1 ) ∈ B ∣ x t = x , α t = α , a t = a ) , B ∈ ℬ ( X × Γ ) , and ( x , α , a ) ∈ K ).

In this manuscript, the transition law Q is assumed to be induced by a system of difference equations, i.e.,

where t = 0 , 1 , … , with ( x 0 , α 0 ) ∈ X × Γ given, where F : K × S 1 → X and G : Γ × S 2 → Γ are measurable functions. Let ε 0 and δ 0 be fixed positive numbers and let ε ∈ [ 0 , ε 0 ] and δ ∈ [ 0 , δ 0 ] and the disturbances { ξ t ( ε ) } and { η t ( δ ) } are sequences of independent and identically distributed (i.i.d.) random elements with values in some Borel spaces ( S 1 , r 1 ) and ( S 2 , r 2 ) , respectively.

In the following, we will consider the space S ≔ S 1 × S 2 with the metric:

for all ( ξ ( ω 1 ) , η ( ω 2 ) ) , ( ξ ˆ ( ω 1 ′ ) , η ˆ ( ω 2 ′ ) ) ∈ S , where ω 1 , ω 1 ′ ∈ Ω 1 and ω 2 , ω 2 ′ ∈ Ω 2 .

Now, consider the random vector χ t ( ε , δ ) ≔ ( ξ t ( ε ) , η t ( δ ) ) for all t ≥ 0 , then the difference equations (1) and (2) can be expressed as follows:

Suppose that there exist s 1 ∈ S 1 and s 2 ∈ S 2 such that s 1 = ξ ( 0 ) and s 2 = η ( 0 ) . Each element of the above sequences depends on numerical parameters ε and δ such that E r 1 ( ξ ( ε ) , s 1 ) → 0 when ε → 0 and E r 2 ( η ( δ ) , s 2 ) → 0 when δ → 0 . On the other hand, a deterministic MDP is considered whose dynamics evolve according to the difference equations shown in equations (3) and (4):

for all t = 0 , 1 , … . Note that χ ( 0 , 0 ) = ( ξ ( 0 ) , η ( 0 ) ) = ( s 1 , s 2 ) , so the joint dynamics given by equations (3) and (4) is denoted as follows:

Under this framework, we are interested in the approximation of MDPs that evolve through (1) and (2) by the deterministic control process given by equations (3) and (4).

When the processes of x -states and α -states are specified by the dynamical model given by equations (1) and (2), the transition law takes the form

where B ∈ ℬ ( X × Γ ) , 1 B ( ⋅ ) denotes the indicator function of B , and μ is the common distribution of the random vector χ t ( ε , δ ) .

On the other hand, when the processes of x -states and α -states are specified by the dynamical model equations (3) and (4), the transition law takes the form

where B ∈ ℬ ( X × Γ ) and ( x , α , a ) ∈ K . Thus, the Markov control model is given by ( X × Γ , A , { A ( x , α ) : ( x , α ) ∈ X × Γ } , Q H , c ) .

A control policy π is a sequence { π t : t = 0 , 1 , … } , where for each t = 0 , 1 , … , π t ( ⋅ ∣ h t ) is a conditional probability on the Borel σ -algebra ℬ ( A ) , given the history h t ≔ ( x 0 , α 0 , a 0 , … , x t − 1 , α t − 1 , a t − 1 , x t , α t ) , such that π t ( A ( x t , α t ) ∣ h t ) = 1 . The set of all policies is denoted by Π .

Let F ≔ { ϕ : X × Γ → A ∣ ϕ is measurable and ϕ ( x , α ) ∈ A ( x , α ) , ( x , α ) ∈ X × Γ } . A sequence π = { ϕ t ∣ t = 0 , 1 , … } of functions ϕ t ∈ F is called a Markov policy. A Markov policy π = { ϕ t ∣ t = 0 , 1 , … } is called a stationary policy if ϕ t = ϕ ∈ F for all t = 0 , 1 , … .

Given initial states ( x 0 = x , α 0 = α ) ∈ X × Γ and arbitrary policy π ∈ Π , there exists a probability measure P ( x , α ) π induced by the triplet ( x , α , π ) over the space Ω = ( X × Γ × A ) ∞ , with ℱ as the σ -algebra product. The existence of this probability measure is verified in an analogous way as in the study by Gonzalez-Hernandez et al. [18]. The corresponding expectation operator is denoted by E ( x , α ) π . The triplet ( x , α , π ) determines a stochastic process ( Ω , ℱ , P ( x , α ) π , { ( x t , α t ) } ) called the Markov decision process. Subsequently, we denote y = ( x , α ) and Y = X × Γ .

3 Problem statement

Consider a deterministic Markov control model ( Y , A , { A ( y ) : y ∈ Y } , Q H , c ) as presented in Section 2. In addition, consider a stochastic control system with the same state space Y , control space A , admissible sets A ( y ) , y ∈ Y , and cost function c , but with the dynamical system described as follows:

Note that when the system is controlled by a deterministic policy, in the stochastic transition law (5), the stochastic system becomes a deterministic system with the transition law (6), when ε → 0 and δ → 0 .

For each policy π ∈ Π and initial state ( x , α ) ∈ Y , consider the expected total discounted cost, denoted by V ˆ ε , δ ( x , α , π ) , and defined as follows:

where β ∈ ( 0 , 1 ) is a discount factor.

Thus, the optimal control problem is to find a policy π * ∈ Π such that

( x , α ) ∈ X × Γ . Then, the optimal value function (optimal cost) is defined as V ε , δ ( x , α ) ≔ inf π ∈ Π V ˆ ε , δ ( x , α , π ) , ( x , α ) ∈ X × Γ . π * is called the optimal policy, while V ε , δ ( x , α ) is called the optimal value function, for ( x , α ) ∈ Y . In the deterministic case, when ε = 0 and δ = 0 , V ε , δ will be denoted by V .

In the next section, we establish conditions to perform an asymptotic analysis of the optimal solution for the stochastic system.

4 Conditions and results

In this section, we introduce three blocks of conditions to study the convergence of the stochastic system defined by equations (1) and (2). In addition, a bound is given for the stability index, which depends on a small noise disturbance parameter δ ˆ ε , δ . In the following, χ ( ε , δ ) denotes a generic element of { χ t ( ε , δ ) } .

Condition 1 is necessary to ensure the existence of minimizers in the corresponding optimality equation. Condition 1(a) is similar to Assumption 1 presented in the study by Gordienko et al. [10].

Let Z : X × Γ → [ 1 , ∞ ) be a measurable function. If U is a real-valued function over X × Γ , then its weighted norm is defined as follows:

where Z denotes the weight function. Let B Z be the Banach space of measurable functions U : Y → R such that ‖ U ‖ Z < ∞ .

Condition 2 is used to provide the existence of solutions of the optimality equation [10]. In addition, under Conditions 1 and 2, the DP approach is valid. Thus, for each ( x , α ) ∈ X × Γ , the following relation holds:

One method of approximating the value function is to use value iterations, which are defined as follows:

where ( x , α ) ∈ X × Γ and n = 1 , 2 , … , with V ε , δ 0 ( ⋅ ) = 0 .

If Conditions 1 and 2 are satisfied, with similar arguments to those in [4] (taking into account the respective changes), the existence of a stationary optimal policy π ε , δ = { f ε , δ , f ε , δ , … } is guaranteed, where f ε , δ ∈ F . Moreover, the following facts hold:

1. V ˆ ε , δ ( x , α , π ε , δ ) = V ε , δ ( x , α ) ∈ B W .

2. E V ε , δ [ H ( y , a , χ ( ε , δ ) ) ] < ∞ , for each ε ∈ [ 0 , ε 0 ] and δ ∈ [ 0 , δ 0 ] , ( y , a ) ∈ K .

Let L be the Kantorovich metric defined on ( S , B s ) :

On the other hand, the stability index Δ ε , δ is defined as follows:

The index Δ ε , δ ( y , π ) expresses the excess of the discounted cost when the policy π is applied to the stochastic control process with respect to equations (1) and (2) for ε , δ > 0 and y ∈ Y . The quality of the approximation for the stochastic system by the policy π 0 * will be measured by the stability index Δ ε , δ ( y , π 0 * ) (see [10,13]), i.e.,

In addition, we define a small-noise disturbance parameter δ ˆ ε , δ as follows:

for ε ∈ [ 0 , ε 0 ] and δ ∈ [ 0 , δ 0 ] . Theorem 1 provides an upper bound for Δ ε , δ ( y , π 0 * ) involving the noise parameter δ ˆ ε , δ .

The following lemmas are applied to prove Theorems 1 and 2.