Humanoid Control

Overview

In this project, I set out to train an agent to control a Unitree G1 model in a simulator to successfully walk. I used the MuJoCo simulation environment and followed the guidelines of "Reinforcement Learning for Versatile, Dynamic, and Robust Bipedal Locomotion Control" (Levine et al.). By the end of this project, the agent was successfully able to stand and partially learned to walk.

Methodology

I used the neural network architecture design described in the paper:

The main component is an MLP, into which is passed a command, reference motion, short history, and a long history that has been fed through a CNN. I did not include reference motions due to time constraints. I used alpha-beta distributions for actuator values as well as control decimation for stability.

The authors of the paper emphasize the importance of both long I/O history and short I/O history: short I/O history provides the current state of the system whereas the long I/O history lets the agent discern mismatches between it's control output and the system's actual behavior. This helps the model counteract unknown environment physics like the friction of its joints, wind, and terrain. As the authors point out, this robustness is especially important when the agent is trained in a simulation because the simulation's dynamics do not perfectly match reality. While I did not intend on deploying this agent to the real world, I still used this model format because, as the authors empirically conclude, having variability in the environment increases the agent's performance compared to having no environment variablity. I chose to include random wind forces to the environment.

I used the MuJoCo physics simulator for training the model. This was my first experience using a physics simulator for robotic control, and I learned a lot about simulations through this project.

I trained the agent using PPO with an entropy bonus.

Experiment Setup and Results

Across the course of this project, I experienced numerous challenges and failures that made me alter or overhaul my plans. Because of this, I cannot state the specific experiment configuration as it was continually changing. So, I will speak generally about the history of my methods over the course of the project.

In both the standing and walking tasks, the robot would begin each episode with its history artificially filled with the standing default configuration.

Standing

My first goal was to get the Unitree robot standing. At first, I was giving the model a reward for having its pelvis at standing height, and a large penalty and an episode restart when it fell. However, the robot was unable to learn despite a lot of hyperparameter tuning:

I tried encoding more information into the reward signal: I gave it a reward proportional to its height. This hilariously led to a "jumping" technique:

The next method I tried was initializing the model's weights to output the standing configuration values. But because the robot started in the stand pose, it was already at maximum reward and so it did not need to learn.

I tried to induce learning by including environment parameters. However, it was incapable of learning to adapt to wind forces, so I concluded this was not a robust method to get the robot to learn tasks.

I finally found some success in a combination of reward signals:

I defined a "fall" as when the robot's pelvis fell below 0.7 meters. Here is a demonstration of a model that learned to stand:

Despite my efforts to get the robot to stand more normally, it would not learn.

Walking

After getting the model to stand, I tried to get it to walk. I used the standing model as a base, and I fed it the walk command and different environment rewards

Training was much harder for the walking command, and despite a lot of reward tuning, I did not see much improvement across training. This is from a checkpoint early in training:

Here is my best model performance:

Future Work

I started this project thinking it would be rather easy to achieve a competent model, but it turned out to be quite difficult. While I got some success in standing and walking, I'm still far from where I hoped I'd be by the end of the summer. However, I had a lot of fun on this project, and I learned a lot about training robots in physics simulators. I look forward to my next project in robotic RL.

Here are some of my ideas on how to improve the performance of my agent:

• Rethink my method of hyperparameter tuning. Systematize tuning by doing small-scale tests across different configurations before honing in on a particular config.
• Incorporate reference motions. I'm sure someone has created walking cycle reference frames for the Unitree robot model in MuJoCo.
• Include more environment variability, and make sure the robot is actively learning the environment from its history feedback.