prePuller.extraTolerations added for the image-puller daemonsets

[jerkern]

Adding exclusive extra tolerations to the prePuller daemonset, to make it easier to integrate into a deployment that uses eg. nidhogg for dynamic tainting.
Related to issue #1836 and #1851 . This doesn't fully solve the problem, it just makes it easier to do a custom solution.

Example use-case

• nidhogg taints all nodes while the prePuller is running, prohibiting anything except the prePuller from being assigned to the node
• prePuller completes and nidhogg removes the taint
• now placeholder and actual user pods can be deployed to the node

This PR accomplishes being able to add a taint that only is tolerated by the prePuller and nothing else.

[welcome]

Thanks for submitting your first pull request! You are awesome! 🤗

If you haven't done so already, check out Jupyter's Code of Conduct. Also, please make sure you followed the pull request template, as this will help us review your contribution more quickly.

You can meet the other Jovyans by joining our Discourse forum. There is also a intro thread there where you can stop by and say Hi! 👋

Welcome to the Jupyter community! 🎉

[consideRatio]

Thanks for your work on this @jerkern! This looks sensible and thank you for providing documentation for everyone else about the motivation of this PR even though we discussed it verbally before.

After these parts are addressed I think we can merge.

• Let the configuration for job.yaml (image-awaiter) in prePuller.hook.tolerations remain
• Add a valid entry in lint-and-validate-config.yaml for the new tolerations, then run tools/templates/lint-and-validate.py to see that we render into valid k8s resources still.

Bonus:

• See if you can rebase on current master to avoid inclusion of unrelated commits in git history.

[donotpush]

@consideRatio we updated the PR based on your comments

[welcome]

Congrats on your first merged pull request in this project! 🎉

Thank you for contributing, we are very proud of you! ❤️

[consideRatio]

Thank you @jerkern and @donotpush! 🎉 ❤️!