摘要
针对深度强化学习算法在复杂动态环境中训练时,由于环境的部分可观测性原因导致智能体难以获得有用信息而不能学习到良好策略且算法收敛速度慢等典型问题,提出一种基于LSTM和非对称actor-critic网络的改进DDPG算法。该算法在actor-critic网络结构中引入LSTM结构,通过记忆推理来学习部分可观测马尔可夫状态中的隐藏状态,同时在actor网络只使用RGB图像作为部分可观测输入的情况下,critic网络利用仿真环境的完全状态进行训练构成非对称网络,加快了训练收敛速度。通过在ROS中进行机械臂抓取仿真实验,结果显示该算法相比于DDPG、PPO和LSTM-DDPG算法获得了更高的成功率,同时具有较快的收敛速度。
When the deep reinforcement learning algorithm is trained in a complex dynamic environment,it is difficult for the agent to obtain useful information due to the partial observability of the environment,which leads to typical problems such as failure to learn good strategies and slow algorithm convergence speed.This paper proposed an improved DDPG algorithm based on LSTM and asymmetric actor-critic network.This method introduced the LSTM structure into actor-critic network to learn the hidden states in partially observable Markov through memory reasoning.At the same time,when the actor network only used RGB images as partially observable inputs,the critic network used the complete state of the simulation environment to train,which formed an asymmetric network and speeded up the training convergence.The simulation experiment of manipulator grasping in ROS shows that the proposed algorithm has higher success rate and faster convergence speed compared with DDPG,PPO and LSTM-DDPG.
作者
何富君
王晓争
刘凯
He Fujun;Wang Xiaozheng;Liu Kai(School of Mechanical Science&Engineering,Northeast Petroleum University,Daqing Heilongjiang 163318,China)
出处
《计算机应用研究》
CSCD
北大核心
2022年第1期183-187,共5页
Application Research of Computers
基金
国家重点研发计划资助项目(2018YFE0196000)
东北石油大学研究生创新科研资助项目(YJSCX2017-022NEPU)。
作者简介
何富君(1970-),男,黑龙江铁力人,教授,博士,主要研究方向为机器人控制技术;通信作者:王晓争(1995-),男,河南许昌人,硕士研究生,主要研究方向为深度强化学习、机器人技术(xiaozhengwang95@163.com);刘凯(1993-),男,河北保定人,助教,硕士,主要研究方向为机器人技术.
