Deep Reinforcement Learning in Immersive Virtual Reality Exergame for Agent Movement Guidance
Immersive Virtual Reality applied to exercise games has a unique potential to both guide and motivate users in performing physical exercise. Advances in modern machine learning open up new opportunities for more significant intelligence in such games. To this end, we investigate the following research question: What if we could train a virtual robot arm to guide us through physical exercises, compete with us, and test out various double-jointed movements? This paper presents a new game mechanic driven by artificial intelligence to visually assist users in their movements through the Unity Game Engine, Unity Ml-Agents, and the HTC Vive Head-Mounted Display. We explore how deep reinforcement learning through Proximal Policy Optimization and Generative Adversarial Imitation Learning can be applied to complete physical exercises from the same immersive virtual reality game. We examine our mechanics with four users through protecting a virtual butterfly with an agent that visually helps users as a cooperative "ghost arm" and an independent competitor. Our results suggest that deep learning agents are effective at learning game exercises and may provide unique insights for users.
It's official, we've been published! More details behind this project can be found at the following manuscript: Elor, A., & Kurniawan, S. (2020, August). Deep Reinforcement Learning in Immersive Virtual Reality Exergame for Agent Movement Guidance. In 2020 IEEE 8th International Conference on Serious Games and Applications for Health(SeGAH). IEEE, 2020.
Feel free to explore this project at: https://github.com/avivelor/UnityMachineLearningForProjectButterfly
Related Materials:
* Virtual Reality Demo (HTC Vive, Stable) - https://github.com/avivelor/UnityMachineLearningForProjectButterfly/raw/master/UnitySDK/ExoButterflyVR-HTCViveBuild.zip
* Standalone Downloadable Demo (Stable) - https://github.com/avivelor/UnityMachineLearningForProjectButterfly/raw/master/UnitySDK/ExoButterfly-StandaloneBuild.zip
* Imitation Learning and Human vs Neural Network Research Video - https://youtu.be/ckMaDXHUGrw
* Early Reinforcement Learning Demo Video - https://youtu.be/5J7xes28bZA
* Blog Posts - https://www.avivelor.com/
External Tools Used and Modified for this Project:
* Unity Machine Learning Agents Beta - https://github.com/Unity-Technologies/ml-agents
* Project Butterfly - https://www.avivelor.com/post/project-butterfly
* Unity ML Agents Introduction - https://towardsdatascience.com/an-introduction-to-unity-ml-agents-6238452fcf4c
* Unity ML Agents Reacher Example - https://github.com/Unity-Technologies/ml-agents/tree/master/Project/Assets/ML-Agents/Examples/Reacher
* Older Unity ML Reacher Example by PHRABAL - https://github.com/PHRABAL/DRL-Reacher
* Proximal Policy Optimization (PPO) in Unity - https://github.com/Unity-Technologies/ml-agents/blob/master/docs/Training-PPO.md
* General Adversarial Imitation Learning (GAIL) in Unity - https://github.com/Unity-Technologies/ml-agents/blob/master/docs/Training-Imitation-Learning.md
* Deep Reinforcement Learning (through Deep Deterministic Policy Gradient or DDPG) - https://arxiv.org/pdf/1509.02971.pdf
* HTC Vive Virtual Reality System (2018) - https://www.vive.com/us/product/vive-virtual-reality-system/
* HTC Vive Trackers (2018) - https://www.vive.com/us/vive-tracker/
Reading References:
* Unity Machine Learning - https://unity3d.com/machine-learning
* Academic Paper on Project Butterfly at IEEEVR 2019 Paper by Elor et Al - https://www.researchgate.net/publication/335194991_Project_Butterfly_Synergizing_Immersive_Virtual_Reality_with_Actuated_Soft_Exosuit_for_Upper-Extremity_Rehabilitation