Reinforcement learning with Gaussian process regression using variational free energy

2.1 Reinforcement learning

In reinforcement learning, the environment is often modeled using the Markov decision process (MDP) [12]. The MDP is represented by M = ⟨ S , A , R , p , γ ⟩ . Here, S and A denote the set of states and actions, R : S → R is a function of immediate reward, p : S × A × S → [ 0 , 1 ] is the transition probability, and γ ∈ [ 0 , 1 ) is the discount rate. Let π : S × A → [ 0 , 1 ] be the action rule of the agent called the policy. We have to estimate the value function from the trajectory { s 0 , a 0 , r 0 , s 1 , a 1 , r 1 , … } generated from the MDP and the policy. We define the discounted reward sum as R t = r t + γ r t + 1 + ⋯ . In reinforcement learning, the goal is to find the action rule that maximizes the reward, i.e., to find π ∗ = max π E π [ R 0 ] , where E π [ ⋅ ] represents the expected value under the given policy π . To find π ∗ , we introduce a value function Q π ( s , a ) = E π [ R 0 ∣ s 0 = s , a 0 = a ] . Since π ∗ can be obtained from the optimal value function Q ∗ = max π Q π ( s , a ) , finding π ∗ reduces to finding Q ∗ . Q-learning [13] is often used as a method to find Q ∗ . If the value function is in table form, we can update the value function with Q t + 1 ( s t , a t ) = Q t ( s t , a t ) + α t [ r t + γ max a Q t ( s t + 1 , a ) − Q t ( s t , a t ) ] . In the case of function approximation, the value of Q ( s t , a t ) is updated to be closer to the value on the right-hand side.

2.2 Gaussian process regression

Gaussian process regression [3] is regarded as a Bayesian nonparametric regression, where we estimate the function y = f ( x ) from the input variable x to the output variable y . Let D = { X , Y } , X = { x n ∈ R p } n = 1 N , and Y = { y n ∈ R } n = 1 N be the data of N pairs of input and output variables. Assume that the output variable is accompanied by normally distributed noise, and the model is y i = f ( x i ) + ε i , ( i = 1 , … , N ) , ε i ∼ iid N ( 0 , σ 2 ) . The goal is to construct the predictive distribution of the output y ∗ for a new input x ∗ . Let f ∼ N ( 0 , K N N ) be the prior distribution. The covariance function is defined as [ K N N ] i , j = k ( x i , x j ) , where k ( ⋅ , ⋅ ) denotes a kernel function. In this article, we use the radial basis function (RBF) kernel k ( x 1 , x 2 ) = α 2 exp [ − ∣ ∣ x 1 − x 2 ∣ ∣ 2 ∕ β 2 ] . The predictive distribution is obtained from the formula for the posterior distribution of the normal distribution. The predictive distribution of y ∗ is

with k ∗ N = ( k ( x ∗ , x 1 ) , … , k ( x ∗ , x N ) ) . Only matrix multiplication is required to calculate the predictive distribution, and the computational cost is O ( N 3 ) . One way to reduce the computational cost is to use inducing points. The partially independent training condition [14], the fully independent training condition [15], and the VFE [11] are well-known methods that use inducing points. In methods using inducing points, the computational cost is O ( N M 2 ) , where M is the number of inducing points, and, in general, N > M .

We introduce the VFE method and explain this method in detail. Let Z = { z j ∈ R p } j = 1 M be the set of inducing points. The model for Gaussian process regression using inducing points is

with u = ( f ( z 1 ) , … , f ( z M ) ) , [ K M M ] i , j = k ( z i , z j ) , and [ K M N ] i , j = k ( z i , x j ) . For details, see [11]. Then, the predictive distribution of y ∗ is

with k ∗ M = ( k ( x ∗ , z 1 ) , … , k ( x ∗ , z M ) ) . In the VFE method, we can estimate the predictive distribution by computing these equations. If we used all data each time for mini-batch learning, the computational cost would be high. Therefore, in the next section, we will extend the method so that it can be estimated without changing the computational cost.

2.3 Related work

Here, we introduce previous research on reinforcement learning using Gaussian process regression. Reinforcement learning algorithms can be divided into two categories: model-based and model-free. In model-based reinforcement learning algorithms, Gaussian process regression can be used not only for the value function but also for the environment model, such as [16–18]. Since the model of the environment is also estimated, the computational cost is higher, but it is not necessary to learn the model of the environment.

Next, we discuss model-free reinforcement learning algorithms. Reinforcement learning algorithms can be categorized into two types: on-policy learning and off-policy learning. SARSA [4] is a typical algorithm for on-policy learning. GP-SARSA [6,7] is an algorithm based on SARSA that learns a value function using a Gaussian process. iGP-SARSA [8] is a method that improves the exploration of GP-SARSA. On the other hand, Q-learning is a typical off-policy learning algorithm. GPQ [9] is a learning method based on Q-learning in which the value function is represented by a Gaussian process.

Next, we describe online learning for Gaussian process regression. GPQ uses the sparse online Gaussian processes method [10] to construct the algorithm. While the Gaussian regression algorithm used by these methods is complex, this article proposes an algorithm that can be updated with only two formulas. The difference between these Gaussian regression methods and the proposed method is the use of inducing points. Offline learning methods for Gaussian process regression with inducing points include VFE [11], fully independent training conditional (FITC) [15], and partially independent training conditional (PITC) [14]. However, reinforcement learning algorithms require online or mini-batch training. Online learning methods have been proposed for FITC and PITC without changing the computational cost [19].

In this article, we use the VFE, which is widely used and can be computed with a simple formula. We also propose mini-batch learning, which has not been proposed in previous studies. The relationship between the previous study and the proposed method is summarized in Table 1. We construct a Q-learning algorithm using our proposed mini-batch VFE. The difference from previous studies is that our proposed mini-batch VFE is used, but the original structure is the same as that of Q-learning.