CI rework - use kind, validate->test->publish, contrib and release rework

[consideRatio]

Review requests

I hope you don't mind I sent a review request. I find lowering the complexity of making code contributions and reducing the complexity of making a release is essential for this repository at this point in time. A concrete suggestion of a review contribution of great value would be to read through CONTRIBUTING.md and RELEASE.md.

PR Changes

I made a huge amount of changes in one go building on @clkao's excellent work with #1259 and #1260 and @manics past excellent work with setting up a VM for local development using a Vagrantfile. Almost all the changes relate to the CI/CD system and contribution related documentation, so while the amount of changes are huge, it should not cause a disruption to merge. But there is one deviation - I bumped from JupyterHub 1.0.0 to a recent master commit stabilizing the hub on startup.

I have so much appreciation for the work of @clkao and @manics that I'm building on top of currently ❤️ 🌻

Local development

I've worked to make local development easy and compatible with the CI runs. I've also made an opinionated switch to kind from minikube. VirtualBox and Vagrantfile are powering the local development as well as local runs of CI tests.

• Both minikube and kind can do the job, but kind supports multiple nodes etc and is embraced by CNCF, so I opted to go all out on kind and drop the outdated documentation on how to use minikube.
• I aligned the documentation to focus on using a VM for local development and added the dev script to help out. Key benefits of this includes: a similar development environment for contributors allowing issues to be easier to reproduce, less dependencies on the operating system as not even python is required, quick to setup and start fresh with little risk of messing with current installation of kubectl etc.
• I bumped to use Ubuntu 18.04 (bionic) instead of 16.04 (xenial) as there was plenty if additional complexity to install Python 3.6 with pip properly on xenial while it worked very smoothly on bionic.

Travis multi-stage builds and tagged releases

The Travis CI/CD pipeline now has three stages, lint and validate -> test -> publish, where publish will only be run on pushes to the master branch.

I've also planned for handling tagged pushes allowing a release process to be a lot easier where we no longer require manual dockerhub and helm chart upload for them.

CONTRIBUTING.md and RELEASE.md rework

I moved RELEASE.md out from the CONTRIBUTING.md documentation and reworked both thoroughly.

In RELEASE.md I made the release issue template contain the relevant details for how to do things, and trimmed away lots of details.

In CONTRIBUTING.md I removed outdated minikube related instructions and focused solely on setting up a VM with the Vagrantfile and developing within the VM. I believe a local development experience should be a lot more consistent and reliable for everyone now, and that it can mimic runs on the CI pipeline as well.

Other

• @clkao's added a fix to stop Travis from fooling chartpress to rebuild all images no matter what, which saves us a lot of CI execution time and delay! Fixes PR Discussion - chartpress to not rebuild all the time #1403
• Added loading of locally build images to the kind cluster, which is required as it is not be able to access the local docker registry. Closes PR Discussion - chartpress and kind #1399.
• Network policies are now enabled by default and tested all the time
• CI test stability improvements, which requires us working on JH master until a JupyterHub release is made to contain wait for proxy before accepting requests jupyterhub#2750. Closes JupyterHub CrashLoopBackOff #493.
• Added some README.md files in various folders to explain what they and their content is about, to help onboard anyone curious about the repo quicker.

TODO

• Await review
• Confirm TravisCI will build on a pushed tag. Yes they do: How to build only pushed tags travis-ci/travis-ci#6893.
• Spellcheck bash scripts as well.
• Document command line options for kind-load-docker-images.py.
• Run black code formatting on kind-load-docker-images.py.
• Combine dev-config-netpol.yaml with dev-config.yaml like @minrk does in enable network policy by default #1271.
• Replace bash scripts with with Python!
• Make it hard to use things in the wrong way.
  • Require KUBECONFIG is set in .env
  • A given kube config can contain multiple clusters and credentials etc, we should enforce a specific context to be specified as well!
    • I learned that there is no such thing as a KUBE_CONTEXT environment variable or KUBE_NAMESPACE variable, as it would lead to too unpredictable behaviors. Perhaps we should ask what context to use and update the .env file with that?
    • kubectl config -> get-contexts, current-context are of relevance, we should not run use-context as that manipulates the kubeconfig though.
  • Ensure we work in the right namespace as well
• Delete replaced bash scripts
• CONTRIBUTING.md - describe VM setup
  • Clarify that the repo folder will be mounted in the VM
• CONTRIBUTING.md - describe OS generic local development
• CONTRIBUTING.md - Rewrite documentation about debugging port-forwarding.
• Figure out and fix calico again after my change
• Make port-forward not fail silently but instead provide relevant information.
• Document the kind load docker-image error.
• Make port-forward a standalone command?
• Don't override system Z2JH_ prefixed environment variables when loading the .env file to allow the CI system to easily customize behavior, but override the others, as they could be set for other reasons than z2jh development in mind, like KUBECONFIG for example.
• Await a chartpress bump so I can test the push to master system properly for tagget commits.

Post merge TODO

Moved to #1447

Integrated PRs

I've picked commits and ideas from the following PRs.
Closes #1259
Closes #1260

[clkao]

hey @consideRatio you know you can use ci/Vagrantfile to simulate the travis env right? ;)

do you want to integrate the netpol related changes into #1259?

[consideRatio]

@clkao I know ;D I didn't get to doing that, mainly because of past experience of running into hurdles getting the VM up and running and knowledge of I want to work on this from multiple computers.

This PR branch builds on the tip of #1259 and contains all its commits (except that I've rebased it on master). I'd like to close #1259 and merge this after documentation/verification/review if that's alright with you @clkao.

[manics]

It'll be a few days before I have time to review this.

[willingc]

Thanks @consideRatio. I did a quick scan of the PR and left a few comments.

Overall, looks good. We may want to run the bash scripts through shellcheck if we don't already do so. Thanks.

[consideRatio]

Wiee, I think I've done all that I feel that I wanted to do within this PR now. I figure we could iterate on the local development instructions etc post-merge now.

The k8s 1.16 version is allowed to fail by configuration.

[clkao]

btw @consideRatio, I tried the new ./dev with kind and it worked great.

[consideRatio]

btw @consideRatio, I tried the new ./dev with kind and it worked great.

Thank you @clkao for your help testing as well as making this happen at all by your initial push to make this work!

❤️