I couldn't find an easy working deployment, so I made my own.
Currently RLCraft version 2.8.1
Uses https://github.com/itzg/docker-minecraft-server, with custom configuration to make Forge work.
Requires Docker and docker-compose.
git clone git@github.com:BernardZhao/rlcraft.git
cd rlcraft
docker-compose up -d