Adding helm tests for production recommendations

[davidjumani]

Description

Adds helm kube e2e tests to mimic the recommendations for production deployments mentioned here

Context

This is to ensure that we test gloo as users would deploy it in production and catch potential issues early

Interesting decisions

N/A

Testing steps

Run the helm kube2e tests

Notes for reviewers

The issue mentioned here needs to be fixed

Checklist:

• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have added tests that prove my fix is effective or that my feature works

BOT NOTES:
resolves #8804

[davidjumani]

Replacing the custom readiness probe test with the Production recommendations tests as the latter tests the former along with other prod settings

[github-actions]

Visit the preview URL for this PR (updated for commit b36cdca):

https://gloo-edge--pr8768-helm-prod-rec-xgga4pu8.web.app

_{(expires Fri, 27 Oct 2023 14:32:15 GMT)}

_{🔥 via Firebase Hosting GitHub Action 🌎}

_{Sign: 77c2b86e287749579b7ff9cadb81e099042ef677}

[solo-changelog-bot]

Issues linked to changelog:
#8804

[sam-heilbron]

I like this and I think it's a nice step towards codifying our production recommendations! For each of these tests, we assert that the components are health (which is good) but not much beyond that. I had kind of envisioned we could also update the other suites (https://github.com/solo-io/gloo/blob/main/test/kube2e/gloo/artifacts/helm.yaml for example) so that when we ran the tests we configured as many production recommendations as well. That way we assert both that everything is healthy AND that the behavior matches what we would expect. What do you think about that?

[davidjumani]

Thanks @sam-heilbron ! I would like to use these same prod recommendations for all the tests but setting settings.disableKubernetesDestinations && global.glooRbac.namespaced (which is the standard in all our tests) leads to panic in gloo (#8801), which is why this was added as a separate test.
Most of the other recommendations such as pod disruption budgets, gzip, etc. have their own tests (albeit not e2e or helm tests). I can add those as well if you think it will be better to test them all together

[davidjumani]

Will create a new PR along with the custom readiness probe fix