Are we listening to SIGTERM? We should!

[consideRatio]

By default, a pod can have 30 seconds on it to shut down by first being asked kindly with SIGTERM on its containers, and then forcefully with SIGKILL.

It is very possible that our code running in a container will ignore SIGTERM and then get forcefully killed by SIGTERM after the terminationGracePeriodSeconds has passed. I think our proxy pod does this currently. With that said, I think the quick fix improvement is to reduce the terminationGracePeriodSeconds to zero as it will be an abrupt kill for the process anyhow. But better is if we can react to SIGTERM no process id #1! So, how do we do this?

Related reading:

• Kubernetes best practices on terminating with grace
• Gracefully stopping docker containers
• Proxy pod that runs CHP: issue PR Discussion - Quick pod relocation, shutdown, and readiness #1406
• JupyterHub listens to SIGINT and SIGTERM
• The jupyterhub/k8s-hub image doesn't because it only provides CMD but not ENTRYPOINT, and neither does its base image ubuntu:18.04, which means it will use bash -c that that will fail to react on SIGTERM.
• The Helm chart hub deployment pod will react to SIGTERM because it use the kubernetes command field which is the same as overriding ENTRYPOINT in a Dockerfile (args in k8s is CMD in Dockerfile).

[consideRatio]

I've learned that docker stop will send a SIGTERM and then later a SIGKILL. Very similar to kubernetes, so we can check if a container is good or bad when it comes to this.

I'm closing this along with jupyterhub/configurable-http-proxy#217 is merged where we fixed the issue for the CHP container of the proxy pod. If the autohttps pod has this fix isn't confirmed to me, but they use dumb-init which is all about considering the signals etc so I figure they do care about that signal.