This project Is an early experiment in trying to achieve some of the goals stated below. Most of them were achieved. There are some useful learnings and code here, which we will likely reuse in a future project or revive this one, but for now, the project is on indefinite ice.
gbuild stands for graph build. Gbuild is a meta build-tool for multi language, module & deployment-target projects, that allows you to find the most effective way to build in parallel using existing language-specific tools, while honouring the implicit dependency graph betweeen modules. Currently parallel execution works, and in the future, we plan to add build caching through a plugin system.
Many software projects today contain multiple languages, modules and deployment targets. This can cause a few problems for teams, such as:
- Naive CI builds that build serially, instead of building the most efficient way in parallel.
- Hodge-podge home-built builds, where glue code is built into the build-tools of one language, to call build tools for other languages.
- Needing to "rebuild the world" on each build, because there is no concept of discovery of which module actually changed (for instance, rebuilding a frontend, when only backend code has changed).
- Local and CI builds duplicate the same functionality
A potential solution to this is adopting a tool like Bazel, which is an excellent solution for monorepos and multi-language builds. But unfortunately, this leads to another set of problems, like a high-learning curve, having to throw away existing build-tools and potentially sacrifice IDE integration with these tools.
gbuild is a meta-build tool, it is intended to work with your existing language specific build tools, such as yarn, npm, sbt, gradle or any others. It is intended as an overlay over these tools for efficient CI builds and local workflows.
What the tool does, is allow you to configure your multi-language, multi-module build as a directed graph of targets. Each target can depend on one or more other targets.
These targets are then bundled into an execution plan, which is the set of targets you want to build for a given purpose. gbuild will look at the targets in the execution plan, analyse their dependencies and then execute as much as possible in parallel for speed and efficiency, while honouring the dependency order of the targets that it has derived.
You can have multiple execution plans, for instance one for ci, one for main-releases, one for local-development etc. The intent is to allow local development to be as much a first class citizen as CI pipelines.
In the future, we intend to add target caching, so as to avoid rebuilding targets unecessarily, if they are unchanged.
gbuild -t [target execution plan] -f [yaml configuration file]
The above defaults -t to "build" and -f to ".gbuild.yml" if not defined
Configuration options should be mostly self-explanatory in the example below.
It is important to note, that while the run block inherits the shell-environment in which gbuild is invoked, the shells themselves run in isolation from each other and can only share files. Any environment variables set in a target will not be available to other targets, or the parent shell.
targets:
- name: Frontend
max_retries: 2
work_dir: front-end/
run:
|-
yarn test
yarn build
- name: Backend
run:
|-
sbt test assembly
- name: PackageFE
work_dir: front-end/
depends_on:
- Frontend
run:
|-
docker build . -t my_image:latest
docker push my_image:latest
- name: PackageBE
depends_on:
- Frontend
run:
|-
docker build . -t my_backend:latest
docker push my_backend:latest
- name: Deploy
depends_on:
- PackageFE
- PackageBE
run:
|-
terraform apply
execution_plans:
- name: CI
targets:
- Frontend
- Backend
- PackageFE
- PackageBE
- Deploy
- name: Local
targets:
- Frontend
- Backend
- Caching of outputs and avoid re-running unchanged targets
- Plugins for cache-storage (local/remote)