Large Language Models (LLMs) have excelled as high-level semantic planners for sequential decision-making tasks. However, harnessing them to learn complex low-level manipulation tasks, such as dexterous pen spinning, remains an open problem. We bridge this fundamental gap and present Eureka, a human-level reward design algorithm powered by LLMs. Eureka exploits the remarkable zero-shot generation, code-writing, and in-context improvement capabilities of state-of-the-art LLMs, such as GPT-4, to perform in-context evolutionary optimization over reward code. The resulting rewards can then be used to acquire complex skills via reinforcement learning. Eureka generates reward functions that outperform expert human-engineered rewards without any task-specific prompting or pre-defined reward templates. In a diverse suite of 29 open-source RL environments that include 10 distinct robot morphologies, Eureka outperforms human expert on 83% of the tasks leading to an average normalized improvement of 52%. The generality of Eureka also enables a new gradient-free approach to reinforcement learning from human feedback (RLHF), readily incorporating human oversight to improve the quality and the safety of the generated rewards in context. Finally, using Eureka rewards in a curriculum learning setting, we demonstrate for the first time a simulated five-finger Shadow Hand capable of performing pen spinning tricks, adeptly manipulating a pen in circles at human speed.
See installation folder for detailed instructions on how to install environments and dependencies.
Below are some example commands to try out Eureka:
python eureka.py env=shadow_hand sample=4 iteration=2 model=gpt-4-0314
python eureka.py env=humanoid sample=16 iteration=5 model=gpt-3.5-turbo-16k-0613
Each run will create a timestamp folder in eureka/outputs that saves the Eureka log as well as all intermediate reward functions and associated policies.
Other command line parameters can be found in eureka/cfg/config.yaml. The list of supported environments can be found in eureka/cfg/env.
We have released Eureka pen spinning policy in isaacgymenvs/isaacgymenvs/checkpoints. Try visualizing it with the following command:
cd isaacgymenvs/isaacgymenvs
python train.py test=True headless=False force_render=True task=ShadowHandSpin checkpoint=checkpoints/EurekaPenSpinning.pth
Note that this script use the default Isaac Gym renderer and not the Omniverse rendering in the paper videos.
- Create a new IsaacGym environment; instructions can be found in here.
- Verify that standard RL works for your new environment.
cd isaacgymenvs/isaacgymenvs
python train.py task=YOUR_NEW_TASK
- Create a new yaml file
your_new_task.yamlineureka/cfg/env:
env_name: your_new_task
task: YOUR_NEW_TASK
description: ...
- Construct the raw environment code that will serve as context for Eureka as well as the skeleton environment code on which the Eureka reward will be appended to:
cd eureka/utils
python prune_env.py your_new_task
- Try out Eureka!
python eureka.py env=your_new_task
We thank the following open-sourced projects:
- Our environments are from IsaacGym and DexterousHands.
- Our RL training code is based on rl_games.
This codebase is released under MIT License.
If you find our work useful, please consider citing us!
@article{ma2023eureka,
title = {Eureka: Human-Level Reward Design via Coding Large Language Models},
author = {Yecheng Jason Ma and William Liang and Guanzhi Wang and De-An Huang and Osbert Bastani and Dinesh Jayaraman and Yuke Zhu and Linxi Fan and Anima Anandkumar},
year = {2023},
journal = {arXiv preprint arXiv: Arxiv-2310.12931}
}Disclaimer: This project is strictly for research purposes, and not an official product from NVIDIA.