Model-Reference Reinforcement Learning Control of Autonomous Surface Vehicles with Uncertainties

2020·Arxiv

Abstract

Abstract

This paper presents a novel model-reference reinforcement learning control method for uncertain autonomous surface vehicles. The proposed control combines a conventional control method with deep reinforcement learning. With the conventional control, we can ensure the learning-based control law provides closed-loop stability for the overall system, and potentially increase the sample efficiency of the deep reinforcement learning. With the reinforcement learning, we can directly learn a control law to compensate for modeling uncertainties. In the proposed control, a nominal system is employed for the design of a baseline control law using a conventional control approach. The nominal system also defines the desired performance for uncertain autonomous vehicles to follow. In comparison with traditional deep reinforcement learning methods, our proposed learning-based control can provide stability guarantees and better sample efficiency. We demonstrate the performance of the new algorithm via extensive simulation results.

I. INTRODUCTION

Autonomous surface vehicles (ASVs) have been attracting more and more attention, due to their advantages in many applications, such as environmental monitoring [1], resource exploration [2], shipping [3], and many more. Successful launch of ASVs in real life requires accurate tracking control along a desired trajectory [4]–[6]. However, accurate tracking control for ASVs is challenging, as ASVs are subject to uncertain nonlinear hydrodynamics and unknown environmental disturbances [7]. Hence, tracking control of highly uncertain ASVs has received extensive research attention [8]–[12].

Control algorithms for uncertain systems including ASVs mainly lie in four categories: 1) robust control which is the “worst-case” design for bounded uncertainties and disturbances [9]; 2) adaptive control which adapts to system uncertainties with parameter estimations [4], [5]; 3) disturbance observer-based control which compensates uncertainties and disturbances in terms of the observation technique [11], [13]; and 4) reinforcement learning (RL) which learns a control law from data samples [12], [14]. The first three algorithms follow a model-based control approach, while the last one is data driven. Model-based control can ensure closed-loop stability, but a system model is indispensable. Uncertainties and disturbances of a system should also satisfy different conditions for different model-based methods. In robust control, uncertainties and disturbances are assumed to be bounded with known boundaries [15]. As a consequence,

Technology, Delft, the Netherlands Qingrui.Zhang@tudelft.nl;

robust control will lead to conservative high-gain control laws which usually limits the control performance (i.e., overshoot, settling time, and stability margins) [16]. Adaptive control can handle varying uncertainties with unknown boundaries, but system uncertainties are assumed to be linearly parameterized with known structure and unknown constant parameters [17], [18]. A valid adaptive control design also requires a system to be persistently excited, resulting in the unpleasant high-frequency oscillation behaviours in control actions [19]. On the other hand, disturbance observer-based control can adapt to both uncertainties and disturbances with unknown structures and without assuming systems to be persistently excited [13], [20]. However, we need the frequency information of uncertainty and disturbance signals when choosing proper gains for the disturbance observer-based control, otherwise it is highly possible to end up with a high-gain control law [20]. In addition, the disturbance observer-based control can only address matched uncertainties and disturbances, which act on systems through the control channel [18], [21]. In general, comprehensive modeling and analysis of systems are essential for all model-based methods.

In comparison with model-based methods, RL is capable of learning a control law from data samples using much less model information [22]. Hence, it is more promising in controlling systems subject to massive uncertainties and disturbances as ASVs [12], [14], [23], [24], given the sufficiency and good quality of collected data. Nevertheless, it is challenging for model-free RL to ensure closed-loop stability, though some research attempts have been made [25]. It implies that the learned control law must be retrained, once some changes happen to the environment or the reference trajectory (i.e. in [14], the authors conducted two independent training procedures for two different reference trajectories.). Model-based RL is possible to learn a control law which ensures the closed-loop stability by introducing a Lyapunov constraint into the objective function of the policy improvement according to the latest research [26]. However, the model-based RL with stability guarantees requires an admissible control law — a control law which makes the original system asymptotically stable — for the initialization. Both the Lyapunov candidate function and complete system dynamics are assumed to be Lipschitz continuous with known Lipschitz constants for the construction of the Lyapunov constraint. It is challenging to find the Lipschitz constant of an uncertain system subject to unknown environmental disturbances. Therefore, the introduced Lyapunov constraint function is restrictive, as it is established based on the worst-case consideration [26].

With the consideration of merits and limitations of existing RL methods, we propose a novel learning-based control algorithm for uncertain ASVs by combining a conventional control method with deep RL in this paper. The proposed learning-based control design, therefore, consists of two components: a baseline control law stabilizing a nominal ASV system and a deep RL control law used to compensate for system uncertainties and disturbances. Such a design method has several advantages over both conventional model-based methods and pure deep RL methods. First of all, in relation to the “model-free” feature of deep RL, we can learn a control law directly to compensate for uncertainties and disturbances without exploiting their structures, boundaries, or frequencies. In the new design, uncertainties and disturbances are not necessarily matched, as deep RL seeks a control law like direct adaptive control [27]. The learning process is performed offline using historical data and the stochastic gradient descent technique, so there is no need for the ASV system be persistently excited when the learned control law is implemented. Second, the overall learned control law can provide stability guarantees, if the baseline control law is able to stabilize the ASV system at least locally. Without introducing a restrictive Lyapunov constraint into the objective function of the policy improvement in RL as in [26], we can avoid exploiting the Lipschitz constant of the overall system and potentially produce less conservative results. Lastly, the proposed design is potentially more sample efficient than a RL algorithm learning from scratch – that is, fewer data samples are needed for the training process. In RL, a system learns from mistakes so a lot of trial and error is demanded. Fortunately, in our proposed design, the baseline control which can stabilize the overall system under no disturbances, can help to exclude unnecessary mistakes, so it provides a good starting point for the RL training. A similar idea is used in [28] for the control of quadrotors. The baseline control in [28] is constructed based on the full accurate model of a quadrotor system, but stability analysis is missing.

The rest of the paper is organized as follows. In Section II, we present the ASV dynamics, basic concepts of reinforcement learning, and problem formulation. Section IV describes the proposed methodology, including deep reinforcement learning design, training setup, and algorithm analysis. In Section VI, numerical simulation results are provided to show the efficiency of the proposed design. Conclusion remarks are given in Section VII.

II. PROBLEM FORMULATION

The full dynamics of autonomous surface vehicles (ASVs) have six degrees of freedom (DOF), including three linear motions and three rotational motions [7]. In most scenarios, we are interested in controlling the horizontal dynamics of (ASVs) [29], [30]. We, therefore, ignore the vertical, rolling, and pitching motions of ASVs by default in this paper.

Let x and y be the horizontal position coordinates of an ASV in the inertial frame and the heading angle as shown in Figure 1. In the body frame (c.f., Figure 1), we use u and v to represent the linear velocities in surge (x-axis) and sway

Fig. 1: Coordinate systems of an autonomous surface vehicle

(y-axis), respectively. The heading angular rate is denoted by r. The general 3-DOF nonlinear dynamics of an ASV can be expressed as

where is a generalized coordinate vector, is the speed vector, M is the inertia matrix, denotes the matrix of Coriolis and centripetal terms, is the damping matrix, represents the control forces and moments, denotes unmodeled dynamics due to gravitational and buoyancy forces and moments [7], and R is a rotation matrix given by

The inertia matrix

where . The matrix

where . The damping matrix

where

, and , and are hydrodynamic coefficients whose definitions can be found in [7]. Accurate numerical models of the nonlinear dynamics (1) are rarely available. Major uncertainty sources come from , and due to hydrodynamics, and due to gravitational and buoyancy forces and moments. The objective of this work is to design a control scheme capable of handling these uncertainties.

Figure 8: Experimental setting CoG for the z-coordinate

Fig. 2: Model-reference reinforcement learning control

III. MODEL-REFERENCE REINFORCEMENT LEARNING CONTROL

where , and . Assume an accurate model (5) is not available, but it is possible to get a nominal model expressed as

where are the known system matrices. Assume that there exists a control law allowing the states of the nominal system (6) to converge to a reference signal

The objective is to design a control law allowing the state of (5) to track state trajectories of the nominal model (6). As shown in Figure 2, the overall control law for the ASV system (5) has the following expression.

where is a baseline control designed based on (6), and a control policy from the deep reinforcement learning module shown in Figure 2. The baseline control is employed to ensure some basic performance, (i.e., local stability), while is introduced to compensate for all system uncertainties. The baseline control in (7) can be designed based on any existing model-based method based on the nominal model (6). Hence, we ignore the design process of , and mainly focus on the development of based on reinforcement learning.

A. Reinforcement learning

In RL, system dynamics are characterized using a Markov decision process denoted by a tuple MDP := is the state space, U specifies the action/input space, defines a transition probability, is a reward function, and is a discount factor. A policy in RL, denoted by , is the probability of choosing an action at a state . Note that the state vector s contains all available signals affecting the reinforcement learning control . In this paper, such signals include , and , where performs like a target state for system (5) and a function of . Hence, we choose

Reinforcement learning uses data samples, so it is assumed that we can sample input and state data from system (5) at discrete time steps. Without loss of generality, we define , and as the ASV state, the baseline control action, and the control action from the reinforcement learning at the time step t, respectively. The state signal s at the time step t is, therefore, denoted by . The sample time step is assumed to be fixed and denoted by

For each state , we define a value function as an expected accumulated return described as

where action-value function (a.k.a., Q-function) is defined to be

In our design, we aim to allow system (5) to track the nominal system (6), so is defined as

where are positive definite matrices. The objective of the reinforcement learning is to find an optimal policy to maximize the state-value function or the action-value function

IV. DEEP REINFORCEMENT LEARNING CONTROL DESIGN

In this section, we will present a deep reinforcement learning algorithm for the design of in (7), where both the control law and the Q-function are approximated using deep neural networks.

The deep reinforcement learning control in this paper is developed based on the soft actor-critic (SAC) algorithm which provides both sample efficient learning and convergence [31]. In SAC, an entropy term is added to the objective function in (11) to regulate the exploration performance at the training stage. The objective of (11) is thus rewritten as

where is an expectation opera- tor, is the entropy of the policy, and is a temperature parameter.

Training of SAC repeatedly executes policy evaluation and policy improvement. In the policy evaluation, a soft Q-value is computed by applying a Bellman operation

Fig. 3: Approximation of

In the policy improvement, the policy is updated by

where denotes the policy from the last update, is the Q-value of denotes the Kullback-Leibler (KL) divergence, and is a normalization factor. Via mathematical manipulations, the objective for the policy improvement is transformed into

More details on how (15) is obtained can be found in [31], [32]. As shown in Figure 3, both the policy value function will be parameterized using fully connected multiple layer perceptrons (MLP) with ’ReLU’ nonlinearities as the activation functions. The ’ReLU’ function is defined as

The “ReLU” activation function outperforms other activation functions like sigmoid functions [33]. For a vector , there exists relu (z) = . Hence, a MLP with ’ReLU’ as the activation functions and one hidden layer is expressed as

whereis a vector composed of z and 1, and and with appropriate dimensions are weight matrices to be trained. For the simplicity, we use to represent the set of parameters to be trained.

Fig. 4: Offline training process of deep reinforcement learning

In this paper, the Q-function is parameterized using and denoted by . The parameterized policy is denoted by , where is the parameter set to be trained. Note that both are a set of parameters whose dimensions are determined by the deep neural network setup. For example, if is represented by a MLP with K hidden layers and L neurons for each hidden layers, the parameter set , and for , where denotes the dimension of the state is the dimension of the input . The deep neural network for is called critic, while the one for is called actor.

A. Training setup

The algorithm training process is illustrated in Figure 4. The whole training process will be offline. We repeatedly run the system (5) under a trajectory tracking task. At each time step t + 1, we collect data samples, such as an input from the last time step , a state from the last time step , a reward , and a current state . Those historical data will be stored as a tuple at a replay memory D [34]. At each policy evaluation or improvement step, we randomly sample a batch of historical data, B, from the replay memory D for the training of the parameters and . Starting the training, we apply the baseline control policy to an ASV system to collect the initial data shown in Algorithm 1. The initial data set is used for the initial fitting of Q-value functions. When the initialization is over, we execute both and the latest updated reinforcement learning policy to run the ASV system.

At the policy evaluation step, the parameters are trained to minimize the following Bellman residual.

where implies that we randomly pick data samples from a replay memory D, and

where is the target parameter which will be updated slowly. Applying a stochastic gradient descent technique (ADAM

[35] in this paper) to (16) on a data batch B with a fixed size, we obtain

where |B| is the batch size.

At the policy improvement step, the objective function defined in (15) is represented using data samples from the replay memory D as given in (17).

Parameter is trained to minimize (17) using a stochastic gradient descent technique. At the training stage, the actor neural network is expressed as

where represents the control law to be implemented in the end, denotes the standard deviation of the exploration noise, is the exploration noise with N (0, I) denoting a Gaussian distribution, and “” is the Hadamard product. Note that the exploration noise is only applied to the training stage. Once the training is done, we only need in the implementation. Hence, at the training stage, in Figure 2 is equal to . Once the training is over, we have

Applying the policy gradient technique to (17), we can calculate the gradient of with respect to in terms

The temperature parameters are updated by minimizing the following objective function.

where is a target entropy. Following the same setting in [32], we choose where “3” here represents the action dimension. In the final implementation, we use two critics which are parameterized by and , respectively. The two critics are introduced to reduce the over-estimation issue in the training of critic neural networks [36]. Under the two-critic mechanism, the target value

The entire algorithm is summarized in Algorithm 1. In Algorithm 1, , and are positive learning rates (scalars), and is a constant scalar.

V. PERFORMANCE ANALYSIS

In this subsection, both the convergence and stability of the proposed learning-based control are analyzed. For the analysis, the soft actor-critic RL method in Algorithm 1 is recapped as a policy iteration (PI) technique which is summarized in Algorithm 2. We thereafter present the following two lemmas without proofs for the convergence analysis [31], [32].

Lemma 1 (Policy evaluation): Let be the Bellman backup operator under a fixed policy . The sequence will converge to the soft Q-function of the policy

Lemma 2 (Policy improvement): Let be an old policy and be a new policy obtained according to (14). There exists

In terms of (1) and (2), we are ready to present Theorem 1 to show the convergence of the SAC algorithm.

Theorem 1 (Convergence): If one repeatedly applies the policy evaluation and policy improvement steps to any control policy , the control policy will converge to an optimal policy denotes a policy set.

Proof: Let be the policy obtained from the i-th policy improvement with . According to Lemma 2, one has , so is monotonically non-decreasing with respect to the policy iteration step i. In addition, is upper bounded according to the definition of the reward given in (10), so will converge to an upper limit , and .

Theorem 1 demonstrates that we can find an optimal policy by repeating the policy evaluation and improvement processes. Next, we will show the closed-loop stability of the overall control law (baseline control plus the learned control The following assumption is made for the baseline control developed using the nominal system (6).

Assumption 1: The baseline control law can ensure that the overall uncertain ASV system is stable – that is, there exists a Lyapunov function associate with such that Note that the baseline control is implicitly included in the state vector s, as s consists of in this paper as discussed in Section III. Hence, in Assumption 1 is the Lyapunov function for the closed-loop system of (5) with the baseline control

Assumption 1 is possible in real world. One could treat the nominal model (6) as a linearized model of the overall ASV system (5) around a certain equilibrium. Therefore, a control law, which ensures asymptotic stability for (6), can ensure at least local stability for (5) [37]. In the stability analysis, we will ignore the entropy term , as it will converge to zero in the end and it is only introduced to regulate the exploration magnitude. Now, we present Theorem 2 to demonstrate the closed-loop stability of the ASV system (5) under the composite control law (7).

Theorem 2 (Stability): Suppose Assumption 1 holds. The overall control law can always stabilize the ASV system (5), where represents the RL control law from i-th iteration, and Proof: In our proposed algorithm, we start the training/learning using the baseline control law . According to Lemma 1, we are able to obtain the corresponding Q value function for the baseline control law . Let the Q value function be where is a function of s. According to the definitions of the reward function in (10) and Q value function in (9), we can choose the Lyapunov function candidate as . If Assumption 1 holds, there exists In the policy improvement, the control law is updated by

where the expectation operator is ignored as the system is deterministic. For any nonlinear system , a necessary condition for the existence of (22) is

Substituting (23) back into (22 yields

Hence, is a control law which can stabilize the same ASV system (5), if Assumption 1 holds. Applying Lemma 1 to we can get a new Lyapunov function . In terms of , (22) and (23), we can show that also stabilizes the ASV system (5). Repeating (22) and (23) for all i = 1, 2, . . ., we can prove that all can stabilize the ASV system (5), if Assumption 1 holds.

VI. SIMULATION

In this section, the proposed learning-based control algo- rithm is implemented to the trajectory tracking control of a supply ship model presented in [29], [30]. Model parameters are summarized in Table I. The unmodeled dynamics in the simulations are given by , , respectively. The based-line control law is designed based on a nominal model with the following simplified linear dynamics in terms of the backstepping control method [13], [37].

where . The reference signal is assumed to be produced by the following motion planner.

where is the generalized reference position vector, is the generalized reference velocity vector, and . In the simulation, the initial position vector is chosen to be , and we set and . The reference acceleration and angular rates are chosen to be

TABLE I: Model parameters

TABLE II: Reinforcement learning configurations

Fig. 5: Learning curves of two RL algorithms at training (One episode is a training trial, and 1000 time steps per episode)

At the training stage, we uniformly randomly sample x (0) and from (0.2, 0.4), and we choose v (0) = 0 and r (0) = 0. The proposed control algorithm is compared with two benchmark designs: the baseline control and the RL control without . Configurations for the training and neural networks are found in Table II. The matrix G and H are chosen to be G = diag {0.025, 0.025, 0.0016, 0.005, 0.001, 0} and H = , respectively.

At the training stage, we run the ASV system for 100 s, and the repeat the training processes for 1000 times (i.e., 1000 episodes). Figure 5 shows the learning curves of the proposed algorithm (red) and the RL algorithm without baseline control (blue). The learning curves demonstrate that both of the two algorithms will converge in terms of the long term returns. However, our proposed algorithm results in a larger return (red) in comparison with the RL without baseline control (blue). Hence, the introduction of the baseline control helps to increase the sample efficiency significantly, as the proposed algorithm (blue) converges faster to a higher return value.

At the first evaluation stage, we run the ASV system for 200 s to demonstrate whether the control law can ensure stable trajectory tracking. Note that we run the ASV for 100 s at training. The trajectory tracking performance of the three algorithms (our proposed algorithm, the baseline control and only RL control) is shown in Figures 6. As observed from Figure 6.b, the control law learned merely using deep RL fails to ensure stable tracking performance. It implies that only deep RL cannot ensure the closed-loop stability. In addition, the baseline control itself fails to achieve acceptable tracking performance mainly due to the existence of system uncertainties. By combining the baseline control and deep RL, the trajectory tracking performance is improved dramatically, and the closed-loop stability is also ensured. The position tracking errors are summarized in Figure 7 and 8. Figure 9 shows the absolute distance errors used to compare the tracking accuracy of the three algorithms. The introduction of the deep RL increases the tracking performance substantially.

At the second evaluation, we still run the ASV system for 200 s, but change the reference trajectory. Note that we use the same learned control laws in both the first and the second evaluations. In the second evaluation, the reference angular acceleration is changed to

The trajectory tracking results are illustrated in Figure 10. Apparently, the proposed control algorithm can ensure closed-loop stability, while the vanilla RL fails to do so. A better tracking performance is obtained by the proposed control law in comparison with only baseline control.

VII. CONCLUSIONS

In this paper, we presented a novel learning-based algorithm for the control of uncertain ASV systems by combining a conventional control method with deep reinforcement learning. With the conventional control, we ensured the overall closed-loop stability of the learning-based control and increase the sample efficiency of the deep RL. With the deep RL, we learned to compensate for the model uncertainties, and thus increased the trajectory tracking performance. In the future works, we will extend the results with the consideration of environmental disturbances. The theoretical results will be further verified via experiments instead of simulations. Sample efficiency of the proposed algorithm will also be analyzed.

REFERENCES

[1] D. O.B.Jones, A. R.Gates, V. A.I.Huvenne, A. B.Phillips, and B. J.Bett, “Autonomous marine environmental monitoring: Application in decommissioned oil fields,” Science of The Total Environment, vol. 668, no. 10, pp. 835– 853, 2019.

[2] J. Majohr and T. Buch, Advances in Unmanned Marine Vehicles. Institution of Engineering and Technology, 2006, ch. Modelling, simulation and control of an autonomous surface marine vehicle for surveying applications Measuring Dolphin MESSIN.

[3] O. Levander, “Autonomous ships on the high seas,” IEEE Spectrum, vol. 54, no. 2, pp. 26 – 31, 2017.

[4] K. Do, Z. Jiang, and J. Pan, “Robust adaptive path following of underactuated ships,” Autonomous Agents and Multi-Agent Systems, vol. 40, no. 6, pp. 929 – 944, Nov. 2004.

[5] K. Do and J. Pan, “Global robust adaptive path following of underactuated ships,” Automatica, vol. 42, no. 10, pp. 1713 – 1722, Oct. 2006.

[6] C. R. Sonnenburg and C. A. Woolsey, “Integrated optimal formation control of multiple unmanned aerial vehicles,” Journal of Field Robotics, vol. 3, no. 30, pp. 371 – 398, May/Jun. 2013.

Fig. 6: Trajectory tracking results of the three algorithms (The first evaluation)

Fig. 7: Position tracking errors (

Fig. 9: Mean absolute distance errors (

[7] T. I. Fossen, Handbook of Marine Craft Hydrodynamics and Motion Control. John Wiley & Sons, Inc., 2011.

[8] R. A. Soltan, H. Ashrafiuon, and K. R. Muske, “State-dependent trajectory planning and tracking control of unmanned surface vessels,” in Proceedings of 2009 American Control Conference. St. Louis, MO, USA: IEEE, Jun. 2009.

[9] R. Yu, Q. Zhu, G. Xia, and Z. Liu, “Sliding mode tracking control of an underactuated surface vessel,” IET Control Theory & Applications, vol. 6, no. 3, pp. 461 – 466, 2012.

[10] N. Wang, J.-C. Sun, M. J. Er, and Y.-C. Liu, “A novel extreme learning control framework of unmanned surface vehicles,” IEEE Transactions on Cybernetics, vol. 46, no. 5, pp. 1106 – 1117, May 2016.

[11] N. Wang, S. Lv, W. Zhang, Z. Liu, and M. J. Er, “Finite-time observer based accurate tracking control of a marine vehicle with complex unknowns,” arXiv preprint arXiv:1711.00832, vol. 145, no. 15, pp. 406 – 415, 2017.

[12] J. Woo, C. Yu, and N. Kim, “Deep reinforcement learning-based controller for path following of an unmanned surface vehicle,” Ocean Engineering, vol. 183, no. 1, pp. 155 – 166, Dec. 2019.

[13] Q. Zhang and H. H. Liu, “UDE-based robust command filtered backstepping control for close formation flight,” IEEE Transactions on Industrial Electronics, vol. 65, no. 11, pp. 8818–8827, Nov. 2018, early access online, March 12, 2018.

[14] W. Shi, S. Song, C. Wu, and C. L. P. Chen, “Multi pseudo q-learning-based deterministic policy gradient for tracking control of autonomous underwater vehicles,” IEEE Transactions on Neural Networks and Learning Systems, vol. 30, no. 12, pp. 3534 – 3546, Dec. 2019.

[15] T. Shen and K. Tamura, “Robust control of uncertain nonlinear system via state feedback,” IEEE Transactions on Automatic Control, vol. 40, no. 4, pp. 766 – 768, Apr. 1995.

[16] X. Liu, H. Su, B. Yao, and J. Chu, “Adaptive robust control of a class of uncertain nonlinear systems with unknown sinusoidal disturbances,” in Proceedings of 2008 47th IEEE Conference on Decision and Control. Cancun, Mexico, USA: IEEE, Dec. 2008.

[17] W. M. Haddad and T. Hayakawa, “Direct adaptive control for nonlinear uncertain systems with exogenous disturbances,” International Journal of Adaptive Control and Signal Processing, vol. 16, no. 2, pp. 151 – 172, Feb. 2002.

[18] Q. Zhang and H. H. Liu, “Aerodynamic model-based robust adaptive control for close formation flight,” Aerospace Science and Technology, vol. 79, pp. 5 – 16, 2018.

[19] P. A. Ioannou and J. Sun, Robust Adaptive Control. Prentice-Hall, Inc., 1996.

[20] B. Zhu, Q. Zhang, and H. H. Liu, “Design and experimental evaluation of robust motion synchronization control for multivehicle system without velocity measurements,” International Journal of Robust and Nonlinear Control, vol. 28, no. 7, pp. 5437 – 5463, 2018.

[21] S. Mondal and hitralekha Mahanta, “Chattering free adaptive multivariable sliding mode controller for systems with matched and mismatched uncertainty,” ISA Transactions, vol. 52, pp. 335 – 341, 2013.

Fig. 10: Trajectory tracking results of the three algorithms (The second evaluation)

[22] R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introductions, 2nd ed. The MIT Press, 2018.

[23] E. Meyer, H. Robinson, A. Rasheed, and O. San, “Taming an autonomous surface vehicle for path following and collision avoidance using deep reinforcement learning,” arXiv preprint arXiv:1912.08578, 2019.

[24] X. Zhou, P. Wu, H. Zhang, W. Guo, and Y. Liu, “Learn to navigate: Cooperative path planning for unmanned surface vehicles using deep reinforcement learning,” IEEE Access, vol. 7, pp. 165 262 – 165 278, Nov. 2019.

[25] M. Han, Y. Tian, L. Zhang, J. Wang, and W. Pan, “Hmodel-free reinforcement learning with robust stability guarantee,” arXiv preprint arXiv:1911.02875, 2019.

[26] F. Berkenkamp, M. Turchetta, A. Schoellig, and A. Krause, “Safe model-based reinforcement learning with stability guarantees,” in Proceedings of the 31st International Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, CA, USA, Dec. 2017, p. 908919.

[27] R. Sutton, A. Barto, and R. Williams, “Reinforcement learning is direct adaptive optimal control,” IEEE Control Systems Magazine, vol. 12, no. 2, pp. 19 – 22, Apr. 1992.

[28] J. Hwangbo, I. Sa, R. Siegwart, and M. Hutter, “Control of a quadrotor with reinforcement learning,” IEEE Robotics and Automation Letters, vol. 2, no. 4, pp. 2096 – 2103, Oct. 2017.

[29] R. Skjetne, T. I. Fossen, and P. V. Kokotovi´c, “Adaptive maneuvering, with experiments, for a model ship in a marine control laboratory,” Mathematics of Operations Research, vol. 41, pp. 289 – 298, 2005.

[30] Z. Peng, D. Wang, T. Li, and Z. Wu, “Leaderless and leader-follower cooperative control of multiple marine surface vehicles with unknown dynamics,” Nonlinear Dynamics, vol. 74, pp. 95 – 106, 2013.

[31] T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine, “Soft actor-critic: Offpolicy maximum entropy deep reinforcement learning with a stochastic actor,” arXiv preprint arXiv:1801.01290, 2018.

[32] T. Haarnoja, K. H. Aurick Zhou, G. Tucker, S. Ha, J. Tan, V. Kumar, H. Zhu, A. Gupta, P. Abbeel, and S. Levine, “Soft actor-critic algorithms and applications,” arXiv preprint arXiv:1812.05905, 2018.

[33] G. E. Dahl, T. N. Sainath, and G. E. Hinton, “Improving deep neural networks for lvcsr using rectified linear units and dropout,” in Proceedings of 2013 IEEE International Conference on Acoustics, Speech and Signal Processing, Vancouver, BC, Canada, May 2013.

[34] V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, C. B. Stig Petersen, A. Sadik, I. Antonoglou, H. King, D. Kumaran, D. Wierstra, S. Legg, and D. Hassabis, “Human-level control through deep reinforcement learning,” Nature, vol. 518, pp. 529–533, Feb. 2015.

[35] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” arXiv preprint arXiv:1412.69801, 2014.

[36] S. Fujimoto, H. van Hoof, and D. Meger, “Addressing function approximation error in actor-critic methods,” arXiv preprint arXiv:1802.09477, 2018.

[37] H. K. Khalil, Nonlinear Systems, 3rd ed. Prentice Hall, 2001.

designed for accessibility and to further open science