Echo in the Steps: Learning Perceptive Humanoid Parkour with Gated Memory

On September 24 a group of researchers posted a preprint on arXiv titled Echo in the Steps: Learning Perceptive Humanoid Parkour with Gated Memory. Its subject is a problem that sounds simple and is not: getting a two-legged robot across ground with only a few safe places to put a foot, using nothing but the depth camera it carries. By the time a foot comes down, the spot it is aiming for has usually left the camera's view, so the robot has to act on something it saw a moment earlier.

The paper's answer is a perception module in two parts. A saliency prior points the system at places where depth changes sharply near the robot, such as the edge of a box or the end of a beam, because that is where footholds begin and end. A gated memory then decides which observations to keep from one frame to the next and which to drop. A further training term, which the authors call an alternation loss, rewards stepping with one leg after the other and discourages hopping on both feet.

The policy was trained in simulation and then run on a Unitree G1 humanoid. The terrains include boxes, stakes, wedges and narrow beams. The authors report an average success rate of 92.7 percent in simulation, higher than the methods they compare against, and say the behaviour carried over to trials on the physical robot. A project page shows video of the runs.

FOOTNOTE borrows the robot, the cameras and the test track, and invents the rest. You run a lab conveyor of thirty-six obstacles with two buttons. A jumps, and a second press in the air jumps again; B slides under a beam, or dives to the floor if you are airborne. Power cells raise a score multiplier, and five of them charge a shield that absorbs one hit. The belt speeds up twice, with a warning beside the robot each time, and three unshielded hits end the attempt.

None of that models the paper's algorithm or what a G1 can physically do: the double jump, the conveyor and the cells are arcade rules. A run takes about forty seconds, and a practice mode teaches the three moves without costing lives. Play in the browser or download the NES ROM.