Reinforcement Q-Learning Control With Reward Shaping Function for Swing Phase Control in a Semi-active Prosthetic Knee

Yonatan Hutabarat, Kittipong Ekkachai, Mitsuhiro Hayashibe, Waree Kongprawechnon

Research output: Contribution to journalArticlepeer-review

Abstract

In this study, we investigated a control algorithm for a semi-active prosthetic knee based on reinforcement learning (RL). Model-free reinforcement Q-learning control with a reward shaping function was proposed as the voltage controller of a magnetorheological damper based on the prosthetic knee. The reward function was designed as a function of the performance index that accounts for the trajectory of the subject-specific knee angle. We compared our proposed reward function to a conventional single reward function under the same random initialization of a Q-matrix. We trained this control algorithm to adapt to several walking speed datasets under one control policy and subsequently compared its performance with that of other control algorithms. The results showed that our proposed reward function performed better than the conventional single reward function in terms of the normalized root mean squared error and also showed a faster convergence trend. Furthermore, our control strategy converged within our desired performance index and could adapt to several walking speeds. Our proposed control structure has also an overall better performance compared to user-adaptive control, while some of its walking speeds performed better than the neural network predictive control from existing studies.

Original languageEnglish
Article number565702
JournalFrontiers in Neurorobotics
Volume14
DOIs
Publication statusPublished - 2020 Nov 26

Keywords

  • magnetorhelogical damper
  • Q-learning
  • reinforcement learning
  • reward shaping
  • semi-active prosthetic knee

ASJC Scopus subject areas

  • Biomedical Engineering
  • Artificial Intelligence

Fingerprint Dive into the research topics of 'Reinforcement Q-Learning Control With Reward Shaping Function for Swing Phase Control in a Semi-active Prosthetic Knee'. Together they form a unique fingerprint.

Cite this