TY - JOUR
T1 - A Survey of Sim-to-Real Transfer Techniques Applied to Reinforcement Learning for Bioinspired Robots
AU - Zhu, Wei
AU - Guo, Xian
AU - Owaki, Dai
AU - Kutsuzawa, Kyo
AU - Hayashibe, Mitsuhiro
N1 - Publisher Copyright:
IEEE
PY - 2021
Y1 - 2021
N2 - The state-of-the-art reinforcement learning (RL) techniques have made innumerable advancements in robot control, especially in combination with deep neural networks (DNNs), known as deep reinforcement learning (DRL). In this article, instead of reviewing the theoretical studies on RL, which were almost fully completed several decades ago, we summarize some state-of-the-art techniques added to commonly used RL frameworks for robot control. We mainly review bioinspired robots (BIRs) because they can learn to locomote or produce natural behaviors similar to animals and humans. With the ultimate goal of practical applications in real world, we further narrow our review scope to techniques that could aid in sim-to-real transfer. We categorized these techniques into four groups: 1) use of accurate simulators; 2) use of kinematic and dynamic models; 3) use of hierarchical and distributed controllers; and 4) use of demonstrations. The purposes of these four groups of techniques are to supply general and accurate environments for RL training, improve sampling efficiency, divide and conquer complex motion tasks and redundant robot structures, and acquire natural skills. We found that, by synthetically using these techniques, it is possible to deploy RL on physical BIRs in actuality.
AB - The state-of-the-art reinforcement learning (RL) techniques have made innumerable advancements in robot control, especially in combination with deep neural networks (DNNs), known as deep reinforcement learning (DRL). In this article, instead of reviewing the theoretical studies on RL, which were almost fully completed several decades ago, we summarize some state-of-the-art techniques added to commonly used RL frameworks for robot control. We mainly review bioinspired robots (BIRs) because they can learn to locomote or produce natural behaviors similar to animals and humans. With the ultimate goal of practical applications in real world, we further narrow our review scope to techniques that could aid in sim-to-real transfer. We categorized these techniques into four groups: 1) use of accurate simulators; 2) use of kinematic and dynamic models; 3) use of hierarchical and distributed controllers; and 4) use of demonstrations. The purposes of these four groups of techniques are to supply general and accurate environments for RL training, improve sampling efficiency, divide and conquer complex motion tasks and redundant robot structures, and acquire natural skills. We found that, by synthetically using these techniques, it is possible to deploy RL on physical BIRs in actuality.
KW - Bioinspired robots
KW - Engines
KW - Kinematics
KW - Legged locomotion
KW - Reinforcement learning
KW - Robots
KW - Task analysis
KW - Training
KW - reinforcement learning (RL)
KW - sim-to-real
KW - transfer techniques.
UR - http://www.scopus.com/inward/record.url?scp=85118610163&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85118610163&partnerID=8YFLogxK
U2 - 10.1109/TNNLS.2021.3112718
DO - 10.1109/TNNLS.2021.3112718
M3 - Article
AN - SCOPUS:85118610163
JO - IEEE Transactions on Neural Networks and Learning Systems
JF - IEEE Transactions on Neural Networks and Learning Systems
SN - 2162-237X
ER -