Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

sentry-nginx Readiness probe failed #1315

Open
MynameisFragile opened this issue Jun 13, 2024 · 4 comments
Open

sentry-nginx Readiness probe failed #1315

MynameisFragile opened this issue Jun 13, 2024 · 4 comments

Comments

@MynameisFragile
Copy link

Nginx can not pass Readiness probe and pod does not start with an error
Readiness probe failed: Get "http://10.8.32.55:8080/auth/login/": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

in nginx pod logs i can endless auth tries:
10.8.32.1 - - [13/Jun/2024:14:54:22 +0000] "GET / HTTP/1.1" 302 0 "-" "kube-probe/1.27" "-"
10.8.32.1 - - [13/Jun/2024:14:54:25 +0000] "GET /auth/login/ HTTP/1.1" 499 0 "http://10.8.32.55:8080/" "kube-probe/1.27" "-"

Never faced this issue before, but this is not the first time I deploy sentry
Why nginx can not auth and pass the readness prod, from where I should start?

Best Regards

@bmassemin
Copy link
Contributor

bmassemin commented Jun 13, 2024

Hi @MynameisFragile

I'm trying to deploy this helm chart the last few days, and I observed the same issue:

  • nginx has a healthcheck on the web pod, and not on itself
  • 499 is a timeout since the web pod is not ready, therefore k8s doesn't route request to web pod
  • the web pod is not ready because the database is not initialized yet
  • the database is not initialized because helm waits for the deployment to succeed to start the db-init
  • the deployment does not succeed because nginx and web are not ready.

my workaround to bootstrap a clean sentry cluster (with a new database):

  • scaledown web and nginx replicas to 0, it will trigger helm hooks since the installation will be successful
  • scale up web and nginx to 1 once the db-init task is completed.

I hope it'll help.

@MynameisFragile
Copy link
Author

@bmassemin Man , U saved my day, thx a lot , I owe u a beer) I did not try u instruction , but I deployed the previous helm chart version and everything is working as expected, so look like las update broke something in a chart structure. Thank u a lot again , god bless u )

@TartanLeGrand
Copy link
Contributor

TartanLeGrand commented Jun 19, 2024

👀 : #1301 & #1313

💡 : bitnami/charts#26326

@TartanLeGrand
Copy link
Contributor

Okay, it's due to the new version of Nginx Bitnami chart, they have upgrade the readiness and the liveness on the chart.

Idea 💡 1:

  • Stop use hooks for the migrations for make Sentry ready

Idea 💡 2:

  • Change the readiness on the Nginx

Idea 💡 3:

- Rollback the version of nginx

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants