With the increasing maturity of multi-UAV technology and its broad applications in scenarios such as UAV roundup tasks,this paper proposes a novel approach to enhance interception efficiency and system robustness by a...With the increasing maturity of multi-UAV technology and its broad applications in scenarios such as UAV roundup tasks,this paper proposes a novel approach to enhance interception efficiency and system robustness by addressing insufficient historical data utilization and inadequate environmental explo-ration.The multi-UAV roundup problem is formulated as a Markov Decision Process(MDP),and an Improved Cross-Entropy Method with Intrinsic Curiosity-enhanced Multi-Agent Twin Delayed Deep Deterministic Policy Gradient(I2C-MATD3)is designed.Specifically,an Improved Cross-Entropy Method(ICEM)based on global elite samples rapidly optimizes training strategies while generating extensive experience for a Multi-Agent Twin Delayed Deep Deterministic Policy Gradient algorithm augmented with intrinsic curiosity rewards(IC-MATD3).In turn,IC-MATD3 guides the optimization direction of ICEM,enabling a synergistic interaction that facilitates effective historical data exploitation and pro-active environmental exploration for UAV agents to accomplish roundup tasks.Experiments in complex scenarios demonstrate that the proposed algorithm achieves superior training efficiency and conver-gence performance compared to state-of-the-art multi-agent reinforcement learning(MARL)methods.Robustness tests and ablation experiments further validate its enhanced generalizability and robustness.展开更多
Deep reinforcement learning algorithms are revolutionizing intelligent decision-making in air combat,drawing widespread attention and extensive research.However,air combat agents trained with these algorithms face sig...Deep reinforcement learning algorithms are revolutionizing intelligent decision-making in air combat,drawing widespread attention and extensive research.However,air combat agents trained with these algorithms face significant challenges,such as limited decision-making capacities due to adversarial training against relatively fixed and singular expert strategies,and a lack of interpretability and reliability in their decisionmaking processes.To tackle these issues,this paper proposes a self-play training mechanism based on policy switching and opponent selection,allowing air combat agents to refine their capabilities via engaging with previous versions of themselves.Additionally,an explainable decision tree model is developed to clarify the decision logic of these agents.Simulations and results demonstrate that the proposed self-play training approach significantly enhances the decision-making abilities of air combat agents,with late-stage agents showing a 38%improvement over early-stage agents in confrontations with an expert strategy.Moreover,the explainable decision tree model effectively elucidates the decision logic and achieves an 86%win rate against the expert strategy,comparable to the 88%win rate of the air combat agents.展开更多
As joint operations have become a key trend in modern military development,unmanned aerial vehicles(UAVs)play an increasingly important role in enhancing the intelligence and responsiveness of combat systems.However,t...As joint operations have become a key trend in modern military development,unmanned aerial vehicles(UAVs)play an increasingly important role in enhancing the intelligence and responsiveness of combat systems.However,the heterogeneity of aircraft,partial observability,and dynamic uncertainty in operational airspace pose significant challenges to autonomous collision avoidance using traditional methods.To address these issues,this paper proposes an adaptive collision avoidance approach for UAVs based on deep reinforcement learning.First,a unified uncertainty model incorporating dynamic wind fields is constructed to capture the complexity of joint operational environments.Then,to effectively handle the heterogeneity between manned and unmanned aircraft and the limitations of dynamic observations,a sector-based partial observation mechanism is designed.A Dynamic Threat Prioritization Assessment algorithm is also proposed to evaluate potential collision threats from multiple dimensions,including time to closest approach,minimum separation distance,and aircraft type.Furthermore,a Hierarchical Prioritized Experience Replay(HPER)mechanism is introduced,which classifies experience samples into high,medium,and low priority levels to preferentially sample critical experiences,thereby improving learning efficiency and accelerating policy convergence.Simulation results show that the proposed HPER-D3QN algorithm outperforms existing methods in terms of learning speed,environmental adaptability,and robustness,significantly enhancing collision avoidance performance and convergence rate.Finally,transfer experiments on a high-fidelity battlefield airspace simulation platform validate the proposed method's deployment potential and practical applicability in complex,real-world joint operational scenarios.展开更多
[Objective]There are several critical challenges in automated safflower harvesting,particularly the inefficiencies in path planning,suboptimal route quality,and limited decision-making capability under dynamic and com...[Objective]There are several critical challenges in automated safflower harvesting,particularly the inefficiencies in path planning,suboptimal route quality,and limited decision-making capability under dynamic and complex environments.To solve these issues,the problem was formulated as a three-dimensional traveling salesman problem and an enhanced reinforcement learning model named actor-critic reinforcement learning pointer network(AC-RL-PtrNet)was proposed,specifically designed for deployment on intelligent safflower picking robots in agricultural settings.[Methods]First,to address the inherent limitations of conventional attention mechanisms in dynamic environments with complex spatial structures,an enhanced attention module was proposed based on the dynamic exponential moving average framework.By combining multi-head attention,spatial distance encoding,and adaptive exponential smoothing,the improved design allowed the model to better capture long-range dependencies and spatial context among safflowers.Meanwhile,to minimize computational cost while preserving inference quality,a structured pruning approach was adopted,which selectively removed redundant connections in the long short-term memory gates and fully connected layers.In parallel,the critic network was redesigned to improve learning stability and accuracy.This was achieved through the inclusion of batch normalization,residual feature aggregation,and a multi-layer value estimation head,all of which contributed to a tighter actorcritic synergy during policy training.[Results and Discussions]To quantitatively assess the impact of each component,ablation experiments were conducted across various configurations.The results confirmed that each module contributed distinct benefits,while their combination yielded the highest improvements in both planning precision and inference efficiency.This coordinated actor-critic design effectively enhanced both trajectory quality and decision stability,which were critical in sequential robotic picking tasks.Experimental results also demonstrated that,compared with traditional swarm intelligence algorithms particle swarm optimization(PSO),ant colony optimization(ACO),and non-dominated sorting genetic algorithm,the proposed AC-RL-PtrNet model achieved a planning time improvement ranging from-2.63%to 61.87%on the 25-target dataset and from 22.93%to 59.1%on the 31-target dataset.Meanwhile,the optimized paths were significantly shortened across different planning instances,indicating robust generalization capability under varied problem scales.Furthermore,field experiments provided concrete validation of the model's practical applicability.When deployed on a mobile picking robot in real safflower fields,the AC-RL-PtrNet achieved a 9.56%reduction in path length and 5.43%time saved for a 25-target picking task,and a 20.17%path reduction and 29.70%time saving for a 31-target scenario involving a different safflower variety.Overall,these results all indicated that the proposed method exhibited significant advantages in enhancing path planning efficiency and optimizing path quality.[Conclusions]This study offers a practical solution for achieving efficient and robust automatic picking by safflower picking robots and provides new insights into solving 3D combinatorial optimization problems.展开更多
Low Earth orbit(LEO)satellite networks exhibit distinct characteristics,e.g.,limited resources of individual satellite nodes and dynamic network topology,which have brought many challenges for routing algorithms.To sa...Low Earth orbit(LEO)satellite networks exhibit distinct characteristics,e.g.,limited resources of individual satellite nodes and dynamic network topology,which have brought many challenges for routing algorithms.To satisfy quality of service(QoS)requirements of various users,it is critical to research efficient routing strategies to fully utilize satellite resources.This paper proposes a multi-QoS information optimized routing algorithm based on reinforcement learning for LEO satellite networks,which guarantees high level assurance demand services to be prioritized under limited satellite resources while considering the load balancing performance of the satellite networks for low level assurance demand services to ensure the full and effective utilization of satellite resources.An auxiliary path search algorithm is proposed to accelerate the convergence of satellite routing algorithm.Simulation results show that the generated routing strategy can timely process and fully meet the QoS demands of high assurance services while effectively improving the load balancing performance of the link.展开更多
The efficiency of carrier-based aircraft support operation scheduling critically impacts aircraft carrier operational effectiveness by determining sortie generation rates,yet faces significant challenges in complex de...The efficiency of carrier-based aircraft support operation scheduling critically impacts aircraft carrier operational effectiveness by determining sortie generation rates,yet faces significant challenges in complex deck environments characterized by resource coupling,dynamic constraints,and highdimensional state-action spaces.Traditional optimization algorithms and vanilla reinforcement learning(RL)struggle with computational inefficiency,sparse rewards,and adaptability to dynamic scenarios,while human expert systems are constrained by the quality of expert knowledge,and poor expert guidance may even have a negative impact.To address these limitations,this paper proposes a human experience-guided actor-critic reinforcement learning framework that synergizes domain expertise with adaptive learning.First,a dynamic Markov decision process(MDP)model is developed to rigorously simulate carrier deck operations,explicitly encoding constraints on positions,resources,and collision avoidance.Building upon this foundation,a human experience database is constructed to enable real-time pattern-matching-based intervention during agent-environment interactions,dynamically correcting wrong actions to avoid catastrophic states while refining exploration efficiency.Finally,the policy and value network objectives are reshaped to incorporate human intent through hybrid reward functions and adaptive guidance weighting,ensuring balanced integration of expert knowledge with RL's exploration capabilities.Extensive simulations across three scenarios demonstrate superior performance compared to state-of-the-art methods and maintain robustness under suboptimal human guidance.These results validate the framework's ability to harmonize human expertise with adaptive learning,offering a practical solution for real-world carriers.展开更多
In order to improve the autonomous ability of unmanned aerial vehicles(UAV)to implement air combat mission,many artificial intelligence-based autonomous air combat maneuver decision-making studies have been carried ou...In order to improve the autonomous ability of unmanned aerial vehicles(UAV)to implement air combat mission,many artificial intelligence-based autonomous air combat maneuver decision-making studies have been carried out,but these studies are often aimed at individual decision-making in 1 v1 scenarios which rarely happen in actual air combat.Based on the research of the 1 v1 autonomous air combat maneuver decision,this paper builds a multi-UAV cooperative air combat maneuver decision model based on multi-agent reinforcement learning.Firstly,a bidirectional recurrent neural network(BRNN)is used to achieve communication between UAV individuals,and the multi-UAV cooperative air combat maneuver decision model under the actor-critic architecture is established.Secondly,through combining with target allocation and air combat situation assessment,the tactical goal of the formation is merged with the reinforcement learning goal of every UAV,and a cooperative tactical maneuver policy is generated.The simulation results prove that the multi-UAV cooperative air combat maneuver decision model established in this paper can obtain the cooperative maneuver policy through reinforcement learning,the cooperative maneuver policy can guide UAVs to obtain the overall situational advantage and defeat the opponents under tactical cooperation.展开更多
Motion planning is critical to realize the autonomous operation of mobile robots.As the complexity and randomness of robot application scenarios increase,the planning capability of the classical hierarchical motion pl...Motion planning is critical to realize the autonomous operation of mobile robots.As the complexity and randomness of robot application scenarios increase,the planning capability of the classical hierarchical motion planners is challenged.With the development of machine learning,the deep reinforcement learning(DRL)-based motion planner has gradually become a research hotspot due to its several advantageous feature.The DRL-based motion planner is model-free and does not rely on the prior structured map.Most importantly,the DRL-based motion planner achieves the unification of the global planner and the local planner.In this paper,we provide a systematic review of various motion planning methods.Firstly,we summarize the representative and state-of-the-art works for each submodule of the classical motion planning architecture and analyze their performance features.Then,we concentrate on summarizing reinforcement learning(RL)-based motion planning approaches,including motion planners combined with RL improvements,map-free RL-based motion planners,and multi-robot cooperative planning methods.Finally,we analyze the urgent challenges faced by these mainstream RLbased motion planners in detail,review some state-of-the-art works for these issues,and propose suggestions for future research.展开更多
In the evolutionary game of the same task for groups,the changes in game rules,personal interests,the crowd size,and external supervision cause uncertain effects on individual decision-making and game results.In the M...In the evolutionary game of the same task for groups,the changes in game rules,personal interests,the crowd size,and external supervision cause uncertain effects on individual decision-making and game results.In the Markov decision framework,a single-task multi-decision evolutionary game model based on multi-agent reinforcement learning is proposed to explore the evolutionary rules in the process of a game.The model can improve the result of a evolutionary game and facilitate the completion of the task.First,based on the multi-agent theory,to solve the existing problems in the original model,a negative feedback tax penalty mechanism is proposed to guide the strategy selection of individuals in the group.In addition,in order to evaluate the evolutionary game results of the group in the model,a calculation method of the group intelligence level is defined.Secondly,the Q-learning algorithm is used to improve the guiding effect of the negative feedback tax penalty mechanism.In the model,the selection strategy of the Q-learning algorithm is improved and a bounded rationality evolutionary game strategy is proposed based on the rule of evolutionary games and the consideration of the bounded rationality of individuals.Finally,simulation results show that the proposed model can effectively guide individuals to choose cooperation strategies which are beneficial to task completion and stability under different negative feedback factor values and different group sizes,so as to improve the group intelligence level.展开更多
To solve the problem of multi-target hunting by an unmanned surface vehicle(USV)fleet,a hunting algorithm based on multi-agent reinforcement learning is proposed.Firstly,the hunting environment and kinematic model wit...To solve the problem of multi-target hunting by an unmanned surface vehicle(USV)fleet,a hunting algorithm based on multi-agent reinforcement learning is proposed.Firstly,the hunting environment and kinematic model without boundary constraints are built,and the criteria for successful target capture are given.Then,the cooperative hunting problem of a USV fleet is modeled as a decentralized partially observable Markov decision process(Dec-POMDP),and a distributed partially observable multitarget hunting Proximal Policy Optimization(DPOMH-PPO)algorithm applicable to USVs is proposed.In addition,an observation model,a reward function and the action space applicable to multi-target hunting tasks are designed.To deal with the dynamic change of observational feature dimension input by partially observable systems,a feature embedding block is proposed.By combining the two feature compression methods of column-wise max pooling(CMP)and column-wise average-pooling(CAP),observational feature encoding is established.Finally,the centralized training and decentralized execution framework is adopted to complete the training of hunting strategy.Each USV in the fleet shares the same policy and perform actions independently.Simulation experiments have verified the effectiveness of the DPOMH-PPO algorithm in the test scenarios with different numbers of USVs.Moreover,the advantages of the proposed model are comprehensively analyzed from the aspects of algorithm performance,migration effect in task scenarios and self-organization capability after being damaged,the potential deployment and application of DPOMH-PPO in the real environment is verified.展开更多
The scale of ground-to-air confrontation task assignments is large and needs to deal with many concurrent task assignments and random events.Aiming at the problems where existing task assignment methods are applied to...The scale of ground-to-air confrontation task assignments is large and needs to deal with many concurrent task assignments and random events.Aiming at the problems where existing task assignment methods are applied to ground-to-air confrontation,there is low efficiency in dealing with complex tasks,and there are interactive conflicts in multiagent systems.This study proposes a multiagent architecture based on a one-general agent with multiple narrow agents(OGMN)to reduce task assignment conflicts.Considering the slow speed of traditional dynamic task assignment algorithms,this paper proposes the proximal policy optimization for task assignment of general and narrow agents(PPOTAGNA)algorithm.The algorithm based on the idea of the optimal assignment strategy algorithm and combined with the training framework of deep reinforcement learning(DRL)adds a multihead attention mechanism and a stage reward mechanism to the bilateral band clipping PPO algorithm to solve the problem of low training efficiency.Finally,simulation experiments are carried out in the digital battlefield.The multiagent architecture based on OGMN combined with the PPO-TAGNA algorithm can obtain higher rewards faster and has a higher win ratio.By analyzing agent behavior,the efficiency,superiority and rationality of resource utilization of this method are verified.展开更多
In this paper, the reinforcement learning method for cooperative multi-agent systems(MAS) with incremental number of agents is studied. The existing multi-agent reinforcement learning approaches deal with the MAS with...In this paper, the reinforcement learning method for cooperative multi-agent systems(MAS) with incremental number of agents is studied. The existing multi-agent reinforcement learning approaches deal with the MAS with a specific number of agents, and can learn well-performed policies. However, if there is an increasing number of agents, the previously learned in may not perform well in the current scenario. The new agents need to learn from scratch to find optimal policies with others,which may slow down the learning speed of the whole team. To solve that problem, in this paper, we propose a new algorithm to take full advantage of the historical knowledge which was learned before, and transfer it from the previous agents to the new agents. Since the previous agents have been trained well in the source environment, they are treated as teacher agents in the target environment. Correspondingly, the new agents are called student agents. To enable the student agents to learn from the teacher agents, we first modify the input nodes of the networks for teacher agents to adapt to the current environment. Then, the teacher agents take the observations of the student agents as input, and output the advised actions and values as supervising information. Finally, the student agents combine the reward from the environment and the supervising information from the teacher agents, and learn the optimal policies with modified loss functions. By taking full advantage of the knowledge of teacher agents, the search space for the student agents will be reduced significantly, which can accelerate the learning speed of the holistic system. The proposed algorithm is verified in some multi-agent simulation environments, and its efficiency has been demonstrated by the experiment results.展开更多
This paper investigates the guidance method based on reinforcement learning(RL)for the coplanar orbital interception in a continuous low-thrust scenario.The problem is formulated into a Markov decision process(MDP)mod...This paper investigates the guidance method based on reinforcement learning(RL)for the coplanar orbital interception in a continuous low-thrust scenario.The problem is formulated into a Markov decision process(MDP)model,then a welldesigned RL algorithm,experience based deep deterministic policy gradient(EBDDPG),is proposed to solve it.By taking the advantage of prior information generated through the optimal control model,the proposed algorithm not only resolves the convergence problem of the common RL algorithm,but also successfully trains an efficient deep neural network(DNN)controller for the chaser spacecraft to generate the control sequence.Numerical simulation results show that the proposed algorithm is feasible and the trained DNN controller significantly improves the efficiency over traditional optimization methods by roughly two orders of magnitude.展开更多
This work proposes a recorded recurrent twin delayed deep deterministic(RRTD3)policy gradient algorithm to solve the challenge of constructing guidance laws for intercepting endoatmospheric maneuvering missiles with u...This work proposes a recorded recurrent twin delayed deep deterministic(RRTD3)policy gradient algorithm to solve the challenge of constructing guidance laws for intercepting endoatmospheric maneuvering missiles with uncertainties and observation noise.The attack-defense engagement scenario is modeled as a partially observable Markov decision process(POMDP).Given the benefits of recurrent neural networks(RNNs)in processing sequence information,an RNN layer is incorporated into the agent’s policy network to alleviate the bottleneck of traditional deep reinforcement learning methods while dealing with POMDPs.The measurements from the interceptor’s seeker during each guidance cycle are combined into one sequence as the input to the policy network since the detection frequency of an interceptor is usually higher than its guidance frequency.During training,the hidden states of the RNN layer in the policy network are recorded to overcome the partially observable problem that this RNN layer causes inside the agent.The training curves show that the proposed RRTD3 successfully enhances data efficiency,training speed,and training stability.The test results confirm the advantages of the RRTD3-based guidance laws over some conventional guidance laws.展开更多
Autonomous umanned aerial vehicle(UAV) manipulation is necessary for the defense department to execute tactical missions given by commanders in the future unmanned battlefield. A large amount of research has been devo...Autonomous umanned aerial vehicle(UAV) manipulation is necessary for the defense department to execute tactical missions given by commanders in the future unmanned battlefield. A large amount of research has been devoted to improving the autonomous decision-making ability of UAV in an interactive environment, where finding the optimal maneuvering decisionmaking policy became one of the key issues for enabling the intelligence of UAV. In this paper, we propose a maneuvering decision-making algorithm for autonomous air-delivery based on deep reinforcement learning under the guidance of expert experience. Specifically, we refine the guidance towards area and guidance towards specific point tasks for the air-delivery process based on the traditional air-to-surface fire control methods.Moreover, we construct the UAV maneuvering decision-making model based on Markov decision processes(MDPs). Specifically, we present a reward shaping method for the guidance towards area and guidance towards specific point tasks using potential-based function and expert-guided advice. The proposed algorithm could accelerate the convergence of the maneuvering decision-making policy and increase the stability of the policy in terms of the output during the later stage of training process. The effectiveness of the proposed maneuvering decision-making policy is illustrated by the curves of training parameters and extensive experimental results for testing the trained policy.展开更多
A novel method was designed to solve reinforcement learning problems with artificial potential field.Firstly a reinforcement learning problem was transferred to a path planning problem by using artificial potential fi...A novel method was designed to solve reinforcement learning problems with artificial potential field.Firstly a reinforcement learning problem was transferred to a path planning problem by using artificial potential field(APF),which was a very appropriate method to model a reinforcement learning problem.Secondly,a new APF algorithm was proposed to overcome the local minimum problem in the potential field methods with a virtual water-flow concept.The performance of this new method was tested by a gridworld problem named as key and door maze.The experimental results show that within 45 trials,good and deterministic policies are found in almost all simulations.In comparison with WIERING's HQ-learning system which needs 20 000 trials for stable solution,the proposed new method can obtain optimal and stable policy far more quickly than HQ-learning.Therefore,the new method is simple and effective to give an optimal solution to the reinforcement learning problem.展开更多
Reinforcement learning has been widely used for mobile robot learning and control. Some progress of this kind of appreaches is surveyed and argued in a new way which emphasizes on different levels of algorithms accord...Reinforcement learning has been widely used for mobile robot learning and control. Some progress of this kind of appreaches is surveyed and argued in a new way which emphasizes on different levels of algorithms according to different complexity of tasks. The central conjecture is that approaches which combine reactive and deliberative control to robotics scale better to complex real-world applications than purely reactive or deliberative ones. This paper describes ha,sic reactive reinforcement learning algorithms and two classes of approaches to achieve deliberation, which are modular methods and hierarchical methods. By combining reactive and deliberative paradigms,the whole system gains advantages from different control levels. The paper gives results of experiments as a case study to verify the effectiveness of the proposed approaches.展开更多
基金the National Key Lab-oratory of Air-based Information Perception and Fusion(Grant No.202510)the Key Research and Development Program of Shaanxi Province(Grant No.2023-GHZD-33)+2 种基金the Fundamental Research Funds for the Central Universities(Grant No.H20250607)the Open Project of the State Key Laboratory of Intelligent Game(Grant No.ZBKF-23-05)the National Nature Science Foundation of China(Grant No.62003267)to provide fund for conducting experiments.
文摘With the increasing maturity of multi-UAV technology and its broad applications in scenarios such as UAV roundup tasks,this paper proposes a novel approach to enhance interception efficiency and system robustness by addressing insufficient historical data utilization and inadequate environmental explo-ration.The multi-UAV roundup problem is formulated as a Markov Decision Process(MDP),and an Improved Cross-Entropy Method with Intrinsic Curiosity-enhanced Multi-Agent Twin Delayed Deep Deterministic Policy Gradient(I2C-MATD3)is designed.Specifically,an Improved Cross-Entropy Method(ICEM)based on global elite samples rapidly optimizes training strategies while generating extensive experience for a Multi-Agent Twin Delayed Deep Deterministic Policy Gradient algorithm augmented with intrinsic curiosity rewards(IC-MATD3).In turn,IC-MATD3 guides the optimization direction of ICEM,enabling a synergistic interaction that facilitates effective historical data exploitation and pro-active environmental exploration for UAV agents to accomplish roundup tasks.Experiments in complex scenarios demonstrate that the proposed algorithm achieves superior training efficiency and conver-gence performance compared to state-of-the-art multi-agent reinforcement learning(MARL)methods.Robustness tests and ablation experiments further validate its enhanced generalizability and robustness.
基金supported by the Joint Funds of the National Natural Science Foundation of China(U2341216).
文摘Deep reinforcement learning algorithms are revolutionizing intelligent decision-making in air combat,drawing widespread attention and extensive research.However,air combat agents trained with these algorithms face significant challenges,such as limited decision-making capacities due to adversarial training against relatively fixed and singular expert strategies,and a lack of interpretability and reliability in their decisionmaking processes.To tackle these issues,this paper proposes a self-play training mechanism based on policy switching and opponent selection,allowing air combat agents to refine their capabilities via engaging with previous versions of themselves.Additionally,an explainable decision tree model is developed to clarify the decision logic of these agents.Simulations and results demonstrate that the proposed self-play training approach significantly enhances the decision-making abilities of air combat agents,with late-stage agents showing a 38%improvement over early-stage agents in confrontations with an expert strategy.Moreover,the explainable decision tree model effectively elucidates the decision logic and achieves an 86%win rate against the expert strategy,comparable to the 88%win rate of the air combat agents.
基金supported by the National Key Research and Development Program of China(No.2022YFB4300902).
文摘As joint operations have become a key trend in modern military development,unmanned aerial vehicles(UAVs)play an increasingly important role in enhancing the intelligence and responsiveness of combat systems.However,the heterogeneity of aircraft,partial observability,and dynamic uncertainty in operational airspace pose significant challenges to autonomous collision avoidance using traditional methods.To address these issues,this paper proposes an adaptive collision avoidance approach for UAVs based on deep reinforcement learning.First,a unified uncertainty model incorporating dynamic wind fields is constructed to capture the complexity of joint operational environments.Then,to effectively handle the heterogeneity between manned and unmanned aircraft and the limitations of dynamic observations,a sector-based partial observation mechanism is designed.A Dynamic Threat Prioritization Assessment algorithm is also proposed to evaluate potential collision threats from multiple dimensions,including time to closest approach,minimum separation distance,and aircraft type.Furthermore,a Hierarchical Prioritized Experience Replay(HPER)mechanism is introduced,which classifies experience samples into high,medium,and low priority levels to preferentially sample critical experiences,thereby improving learning efficiency and accelerating policy convergence.Simulation results show that the proposed HPER-D3QN algorithm outperforms existing methods in terms of learning speed,environmental adaptability,and robustness,significantly enhancing collision avoidance performance and convergence rate.Finally,transfer experiments on a high-fidelity battlefield airspace simulation platform validate the proposed method's deployment potential and practical applicability in complex,real-world joint operational scenarios.
基金Natural Science Foundation of Xinjiang Uygur Autonomous Region,China Under Grant(2023D01C190)National Science and Technology Major Project(2022ZD0115801)。
文摘[Objective]There are several critical challenges in automated safflower harvesting,particularly the inefficiencies in path planning,suboptimal route quality,and limited decision-making capability under dynamic and complex environments.To solve these issues,the problem was formulated as a three-dimensional traveling salesman problem and an enhanced reinforcement learning model named actor-critic reinforcement learning pointer network(AC-RL-PtrNet)was proposed,specifically designed for deployment on intelligent safflower picking robots in agricultural settings.[Methods]First,to address the inherent limitations of conventional attention mechanisms in dynamic environments with complex spatial structures,an enhanced attention module was proposed based on the dynamic exponential moving average framework.By combining multi-head attention,spatial distance encoding,and adaptive exponential smoothing,the improved design allowed the model to better capture long-range dependencies and spatial context among safflowers.Meanwhile,to minimize computational cost while preserving inference quality,a structured pruning approach was adopted,which selectively removed redundant connections in the long short-term memory gates and fully connected layers.In parallel,the critic network was redesigned to improve learning stability and accuracy.This was achieved through the inclusion of batch normalization,residual feature aggregation,and a multi-layer value estimation head,all of which contributed to a tighter actorcritic synergy during policy training.[Results and Discussions]To quantitatively assess the impact of each component,ablation experiments were conducted across various configurations.The results confirmed that each module contributed distinct benefits,while their combination yielded the highest improvements in both planning precision and inference efficiency.This coordinated actor-critic design effectively enhanced both trajectory quality and decision stability,which were critical in sequential robotic picking tasks.Experimental results also demonstrated that,compared with traditional swarm intelligence algorithms particle swarm optimization(PSO),ant colony optimization(ACO),and non-dominated sorting genetic algorithm,the proposed AC-RL-PtrNet model achieved a planning time improvement ranging from-2.63%to 61.87%on the 25-target dataset and from 22.93%to 59.1%on the 31-target dataset.Meanwhile,the optimized paths were significantly shortened across different planning instances,indicating robust generalization capability under varied problem scales.Furthermore,field experiments provided concrete validation of the model's practical applicability.When deployed on a mobile picking robot in real safflower fields,the AC-RL-PtrNet achieved a 9.56%reduction in path length and 5.43%time saved for a 25-target picking task,and a 20.17%path reduction and 29.70%time saving for a 31-target scenario involving a different safflower variety.Overall,these results all indicated that the proposed method exhibited significant advantages in enhancing path planning efficiency and optimizing path quality.[Conclusions]This study offers a practical solution for achieving efficient and robust automatic picking by safflower picking robots and provides new insights into solving 3D combinatorial optimization problems.
基金National Key Research and Development Program(2021YFB2900604)。
文摘Low Earth orbit(LEO)satellite networks exhibit distinct characteristics,e.g.,limited resources of individual satellite nodes and dynamic network topology,which have brought many challenges for routing algorithms.To satisfy quality of service(QoS)requirements of various users,it is critical to research efficient routing strategies to fully utilize satellite resources.This paper proposes a multi-QoS information optimized routing algorithm based on reinforcement learning for LEO satellite networks,which guarantees high level assurance demand services to be prioritized under limited satellite resources while considering the load balancing performance of the satellite networks for low level assurance demand services to ensure the full and effective utilization of satellite resources.An auxiliary path search algorithm is proposed to accelerate the convergence of satellite routing algorithm.Simulation results show that the generated routing strategy can timely process and fully meet the QoS demands of high assurance services while effectively improving the load balancing performance of the link.
基金supported by funding from the National Natural Science Foundation of China(Grant Nos.62325602,62406292,62302459,62406293,and 62036010)。
文摘The efficiency of carrier-based aircraft support operation scheduling critically impacts aircraft carrier operational effectiveness by determining sortie generation rates,yet faces significant challenges in complex deck environments characterized by resource coupling,dynamic constraints,and highdimensional state-action spaces.Traditional optimization algorithms and vanilla reinforcement learning(RL)struggle with computational inefficiency,sparse rewards,and adaptability to dynamic scenarios,while human expert systems are constrained by the quality of expert knowledge,and poor expert guidance may even have a negative impact.To address these limitations,this paper proposes a human experience-guided actor-critic reinforcement learning framework that synergizes domain expertise with adaptive learning.First,a dynamic Markov decision process(MDP)model is developed to rigorously simulate carrier deck operations,explicitly encoding constraints on positions,resources,and collision avoidance.Building upon this foundation,a human experience database is constructed to enable real-time pattern-matching-based intervention during agent-environment interactions,dynamically correcting wrong actions to avoid catastrophic states while refining exploration efficiency.Finally,the policy and value network objectives are reshaped to incorporate human intent through hybrid reward functions and adaptive guidance weighting,ensuring balanced integration of expert knowledge with RL's exploration capabilities.Extensive simulations across three scenarios demonstrate superior performance compared to state-of-the-art methods and maintain robustness under suboptimal human guidance.These results validate the framework's ability to harmonize human expertise with adaptive learning,offering a practical solution for real-world carriers.
基金Supported by National Natural Science Foundation of China（60474035）,National Research Foundation for the Doctoral Program of Higher Education of China（20050359004）,Natural Science Foundation of Anhui Province（070412035）
基金supported by the Aeronautical Science Foundation of China(2017ZC53033)the Seed Foundation of Innovation and Creation for Graduate Students in Northwestern Polytechnical University(CX2020156)。
文摘In order to improve the autonomous ability of unmanned aerial vehicles(UAV)to implement air combat mission,many artificial intelligence-based autonomous air combat maneuver decision-making studies have been carried out,but these studies are often aimed at individual decision-making in 1 v1 scenarios which rarely happen in actual air combat.Based on the research of the 1 v1 autonomous air combat maneuver decision,this paper builds a multi-UAV cooperative air combat maneuver decision model based on multi-agent reinforcement learning.Firstly,a bidirectional recurrent neural network(BRNN)is used to achieve communication between UAV individuals,and the multi-UAV cooperative air combat maneuver decision model under the actor-critic architecture is established.Secondly,through combining with target allocation and air combat situation assessment,the tactical goal of the formation is merged with the reinforcement learning goal of every UAV,and a cooperative tactical maneuver policy is generated.The simulation results prove that the multi-UAV cooperative air combat maneuver decision model established in this paper can obtain the cooperative maneuver policy through reinforcement learning,the cooperative maneuver policy can guide UAVs to obtain the overall situational advantage and defeat the opponents under tactical cooperation.
基金supported by the National Natural Science Foundation of China (62173251)the“Zhishan”Scholars Programs of Southeast University+1 种基金the Fundamental Research Funds for the Central UniversitiesShanghai Gaofeng&Gaoyuan Project for University Academic Program Development (22120210022)
文摘Motion planning is critical to realize the autonomous operation of mobile robots.As the complexity and randomness of robot application scenarios increase,the planning capability of the classical hierarchical motion planners is challenged.With the development of machine learning,the deep reinforcement learning(DRL)-based motion planner has gradually become a research hotspot due to its several advantageous feature.The DRL-based motion planner is model-free and does not rely on the prior structured map.Most importantly,the DRL-based motion planner achieves the unification of the global planner and the local planner.In this paper,we provide a systematic review of various motion planning methods.Firstly,we summarize the representative and state-of-the-art works for each submodule of the classical motion planning architecture and analyze their performance features.Then,we concentrate on summarizing reinforcement learning(RL)-based motion planning approaches,including motion planners combined with RL improvements,map-free RL-based motion planners,and multi-robot cooperative planning methods.Finally,we analyze the urgent challenges faced by these mainstream RLbased motion planners in detail,review some state-of-the-art works for these issues,and propose suggestions for future research.
基金supported by the National Key R&D Program of China(2017YFB1400105).
文摘In the evolutionary game of the same task for groups,the changes in game rules,personal interests,the crowd size,and external supervision cause uncertain effects on individual decision-making and game results.In the Markov decision framework,a single-task multi-decision evolutionary game model based on multi-agent reinforcement learning is proposed to explore the evolutionary rules in the process of a game.The model can improve the result of a evolutionary game and facilitate the completion of the task.First,based on the multi-agent theory,to solve the existing problems in the original model,a negative feedback tax penalty mechanism is proposed to guide the strategy selection of individuals in the group.In addition,in order to evaluate the evolutionary game results of the group in the model,a calculation method of the group intelligence level is defined.Secondly,the Q-learning algorithm is used to improve the guiding effect of the negative feedback tax penalty mechanism.In the model,the selection strategy of the Q-learning algorithm is improved and a bounded rationality evolutionary game strategy is proposed based on the rule of evolutionary games and the consideration of the bounded rationality of individuals.Finally,simulation results show that the proposed model can effectively guide individuals to choose cooperation strategies which are beneficial to task completion and stability under different negative feedback factor values and different group sizes,so as to improve the group intelligence level.
基金financial support from National Natural Science Foundation of China(Grant No.61601491)Natural Science Foundation of Hubei Province,China(Grant No.2018CFC865)Military Research Project of China(-Grant No.YJ2020B117)。
文摘To solve the problem of multi-target hunting by an unmanned surface vehicle(USV)fleet,a hunting algorithm based on multi-agent reinforcement learning is proposed.Firstly,the hunting environment and kinematic model without boundary constraints are built,and the criteria for successful target capture are given.Then,the cooperative hunting problem of a USV fleet is modeled as a decentralized partially observable Markov decision process(Dec-POMDP),and a distributed partially observable multitarget hunting Proximal Policy Optimization(DPOMH-PPO)algorithm applicable to USVs is proposed.In addition,an observation model,a reward function and the action space applicable to multi-target hunting tasks are designed.To deal with the dynamic change of observational feature dimension input by partially observable systems,a feature embedding block is proposed.By combining the two feature compression methods of column-wise max pooling(CMP)and column-wise average-pooling(CAP),observational feature encoding is established.Finally,the centralized training and decentralized execution framework is adopted to complete the training of hunting strategy.Each USV in the fleet shares the same policy and perform actions independently.Simulation experiments have verified the effectiveness of the DPOMH-PPO algorithm in the test scenarios with different numbers of USVs.Moreover,the advantages of the proposed model are comprehensively analyzed from the aspects of algorithm performance,migration effect in task scenarios and self-organization capability after being damaged,the potential deployment and application of DPOMH-PPO in the real environment is verified.
基金the Project of National Natural Science Foundation of China(Grant No.62106283)the Project of National Natural Science Foundation of China(Grant No.72001214)to provide fund for conducting experimentsthe Project of Natural Science Foundation of Shaanxi Province(Grant No.2020JQ-484)。
文摘The scale of ground-to-air confrontation task assignments is large and needs to deal with many concurrent task assignments and random events.Aiming at the problems where existing task assignment methods are applied to ground-to-air confrontation,there is low efficiency in dealing with complex tasks,and there are interactive conflicts in multiagent systems.This study proposes a multiagent architecture based on a one-general agent with multiple narrow agents(OGMN)to reduce task assignment conflicts.Considering the slow speed of traditional dynamic task assignment algorithms,this paper proposes the proximal policy optimization for task assignment of general and narrow agents(PPOTAGNA)algorithm.The algorithm based on the idea of the optimal assignment strategy algorithm and combined with the training framework of deep reinforcement learning(DRL)adds a multihead attention mechanism and a stage reward mechanism to the bilateral band clipping PPO algorithm to solve the problem of low training efficiency.Finally,simulation experiments are carried out in the digital battlefield.The multiagent architecture based on OGMN combined with the PPO-TAGNA algorithm can obtain higher rewards faster and has a higher win ratio.By analyzing agent behavior,the efficiency,superiority and rationality of resource utilization of this method are verified.
基金supported by the National Key R&D Program of China (2018AAA0101400)the National Natural Science Foundation of China (62173251+3 种基金61921004U1713209)the Natural Science Foundation of Jiangsu Province of China (BK20202006)the Guangdong Provincial Key Laboratory of Intelligent Decision and Cooperative Control。
文摘In this paper, the reinforcement learning method for cooperative multi-agent systems(MAS) with incremental number of agents is studied. The existing multi-agent reinforcement learning approaches deal with the MAS with a specific number of agents, and can learn well-performed policies. However, if there is an increasing number of agents, the previously learned in may not perform well in the current scenario. The new agents need to learn from scratch to find optimal policies with others,which may slow down the learning speed of the whole team. To solve that problem, in this paper, we propose a new algorithm to take full advantage of the historical knowledge which was learned before, and transfer it from the previous agents to the new agents. Since the previous agents have been trained well in the source environment, they are treated as teacher agents in the target environment. Correspondingly, the new agents are called student agents. To enable the student agents to learn from the teacher agents, we first modify the input nodes of the networks for teacher agents to adapt to the current environment. Then, the teacher agents take the observations of the student agents as input, and output the advised actions and values as supervising information. Finally, the student agents combine the reward from the environment and the supervising information from the teacher agents, and learn the optimal policies with modified loss functions. By taking full advantage of the knowledge of teacher agents, the search space for the student agents will be reduced significantly, which can accelerate the learning speed of the holistic system. The proposed algorithm is verified in some multi-agent simulation environments, and its efficiency has been demonstrated by the experiment results.
基金supported by the National Defense Science and Technology Innovation(18-163-15-LZ-001-004-13).
文摘This paper investigates the guidance method based on reinforcement learning(RL)for the coplanar orbital interception in a continuous low-thrust scenario.The problem is formulated into a Markov decision process(MDP)model,then a welldesigned RL algorithm,experience based deep deterministic policy gradient(EBDDPG),is proposed to solve it.By taking the advantage of prior information generated through the optimal control model,the proposed algorithm not only resolves the convergence problem of the common RL algorithm,but also successfully trains an efficient deep neural network(DNN)controller for the chaser spacecraft to generate the control sequence.Numerical simulation results show that the proposed algorithm is feasible and the trained DNN controller significantly improves the efficiency over traditional optimization methods by roughly two orders of magnitude.
基金supported by the National Natural Science Foundation of China(Grant No.12072090)。
文摘This work proposes a recorded recurrent twin delayed deep deterministic(RRTD3)policy gradient algorithm to solve the challenge of constructing guidance laws for intercepting endoatmospheric maneuvering missiles with uncertainties and observation noise.The attack-defense engagement scenario is modeled as a partially observable Markov decision process(POMDP).Given the benefits of recurrent neural networks(RNNs)in processing sequence information,an RNN layer is incorporated into the agent’s policy network to alleviate the bottleneck of traditional deep reinforcement learning methods while dealing with POMDPs.The measurements from the interceptor’s seeker during each guidance cycle are combined into one sequence as the input to the policy network since the detection frequency of an interceptor is usually higher than its guidance frequency.During training,the hidden states of the RNN layer in the policy network are recorded to overcome the partially observable problem that this RNN layer causes inside the agent.The training curves show that the proposed RRTD3 successfully enhances data efficiency,training speed,and training stability.The test results confirm the advantages of the RRTD3-based guidance laws over some conventional guidance laws.
基金supported by the Key Research and Development Program of Shaanxi (2022GXLH-02-09)the Aeronautical Science Foundation of China (20200051053001)the Natural Science Basic Research Program of Shaanxi (2020JM-147)。
文摘Autonomous umanned aerial vehicle(UAV) manipulation is necessary for the defense department to execute tactical missions given by commanders in the future unmanned battlefield. A large amount of research has been devoted to improving the autonomous decision-making ability of UAV in an interactive environment, where finding the optimal maneuvering decisionmaking policy became one of the key issues for enabling the intelligence of UAV. In this paper, we propose a maneuvering decision-making algorithm for autonomous air-delivery based on deep reinforcement learning under the guidance of expert experience. Specifically, we refine the guidance towards area and guidance towards specific point tasks for the air-delivery process based on the traditional air-to-surface fire control methods.Moreover, we construct the UAV maneuvering decision-making model based on Markov decision processes(MDPs). Specifically, we present a reward shaping method for the guidance towards area and guidance towards specific point tasks using potential-based function and expert-guided advice. The proposed algorithm could accelerate the convergence of the maneuvering decision-making policy and increase the stability of the policy in terms of the output during the later stage of training process. The effectiveness of the proposed maneuvering decision-making policy is illustrated by the curves of training parameters and extensive experimental results for testing the trained policy.
基金Projects(30270496,60075019,60575012)supported by the National Natural Science Foundation of China
文摘A novel method was designed to solve reinforcement learning problems with artificial potential field.Firstly a reinforcement learning problem was transferred to a path planning problem by using artificial potential field(APF),which was a very appropriate method to model a reinforcement learning problem.Secondly,a new APF algorithm was proposed to overcome the local minimum problem in the potential field methods with a virtual water-flow concept.The performance of this new method was tested by a gridworld problem named as key and door maze.The experimental results show that within 45 trials,good and deterministic policies are found in almost all simulations.In comparison with WIERING's HQ-learning system which needs 20 000 trials for stable solution,the proposed new method can obtain optimal and stable policy far more quickly than HQ-learning.Therefore,the new method is simple and effective to give an optimal solution to the reinforcement learning problem.
文摘Reinforcement learning has been widely used for mobile robot learning and control. Some progress of this kind of appreaches is surveyed and argued in a new way which emphasizes on different levels of algorithms according to different complexity of tasks. The central conjecture is that approaches which combine reactive and deliberative control to robotics scale better to complex real-world applications than purely reactive or deliberative ones. This paper describes ha,sic reactive reinforcement learning algorithms and two classes of approaches to achieve deliberation, which are modular methods and hierarchical methods. By combining reactive and deliberative paradigms,the whole system gains advantages from different control levels. The paper gives results of experiments as a case study to verify the effectiveness of the proposed approaches.
