Popper

Popper is a tool for defining and executing container-native workflows in Docker, as well as other container engines. With Popper, you define a workflow in a YAML file, and then execute it with a single command. A workflow file looks like this:

version: '1'
steps:
- uses: docker://byrnedo/alpine-curl:0.1.8
  args: [-LO, https://github.com/datasets/co2-fossil-global/raw/master/global.csv]

- uses: docker://python:3.8.1-alpine
  args: [scripts/get_mean.py, global.csv, 'Per Capita']

Assuming the above is stored in a wf.yml file, the workflow gets executed by running:

popper run -f wf.yml

Keep reading down to find installation instructions. The full example above can be found here. For more information on the YAML syntax, see here.

The high-level goals of this project are to provide:

• Lightweight workflow definition syntax. Defining a workflow is as simple as writing file in a lightweight YAML syntax and invoking popper run (see demo above). If you're familiar with Docker Compose, you can think of Popper as Compose but for workflows instead of services.
• An abstraction over container runtimes. In addition to Docker, Popper can seamlessly execute workflows in other runtimes by interacting with distinct container engines. Popper currently supports Singularity and we are working on adding Podman.
• Run on resource managers. Popper can also execute workflows on a variety of resource managers and schedulers such as Kubernetes and SLURM, without requiring any modifications to a workflow YAML file. We currently support SLURM and are working on adding support for Kubernetes.
• Continuous integration. Generate configuration files for distinct CI services, allowing users to run the exact same workflows they run locally on Travis, Jenkins, Gitlab, Circle and others.
• Workflow development. Aid in the implementation and debugging of workflows, and provide with an extensive list of example workflows that can serve as a starting point.

---

This repository contains:

• cli/. The codebase of the command-line tool.
• docs/. General documentation containing guides, CLI documentation and pointers to other resources.
• gh-pages branch. Contents of our landing page.

Installation

To run workflows, you need to have Python 3.6+, Git and a container engine installed (Docker and Singularity are currently supported). To install Popper you can use pip. We recommend to install in a virtual environment (see here for more on virtualenv). To install:

pip install popper

Once installed, you can get an overview and list of available commands:

popper --help

For a Quickstart guide on how to use Popper, look here.

Contributing

Anyone is welcome to contribute to Popper! To get started, take a look at our contributing guidelines, then dive in with our list of good first issues.

Participation Guidelines

Popper adheres to the code of conduct posted in this repository. By participating or contributing to Popper, you're expected to uphold this code. If you encounter unacceptable behavior, please immediately email us.

How to Cite Popper

Ivo Jimenez, Michael Sevilla, Noah Watkins, Carlos Maltzahn, Jay Lofstead, Kathryn Mohror, Andrea Arpaci-Dusseau and Remzi Arpaci-Dusseau. The Popper Convention: Making Reproducible Systems Evaluation Practical. In 2017 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW), 1561–70, 2017. https://doi.org/10.1109/IPDPSW.2017.157.

For BibTeX, click here.