Disable postgres SSL to fix random query timeout

[xin-hedera]

Description:

This PR fixes the random readonly query timeout issue in citus

• Disable SSL
• Enable postgres-util container
• Reduce postgresql work_mem to 24MB to lower memory pressue
• Add TEST_REPORTS_DIR env variable to all testkube tests

Related issue(s):

Fixes #8748
Fixes #9143

Notes for reviewer:

After much effort to diagnose and experiment, found disabling SSL solves the issue. Though I'm not clear about why SSL connection (via unix socket) can cause such issue, the changes here are much needed so we can move on to other tasks.

If needed, we can open a ticket upstream, however, the configuration itself (coordinator -> worker connection pgbouncer pooling) isn't supported by stackgres.

The changes which enable postgres-util container is a must to gain pgbouncer info. Without it, it's impossible to view a lot of pgbouner stats. The problem is pgbouner can only be connected via unix socket with psql command, psql -p 6432 -U pgbouncer pgbouncer. We can do so in the patroni container. However a lot of commands will just fail to show results, e.g., show clients, complaining about more is not available as it's not installed in the patroni container.

Verified for the upgrade path from <= 0.112.0-rc3 helm charts with SSL defaults to enabled, to charts in this PR, no manual steps are needed.

Checklist

• Documented (Code comments, README, etc.)
• Tested (unit, integration, etc.)

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 92.54%. Comparing base (ca7e24b) to head (c7a37ee).
Report is 27 commits behind head on main.

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #9144      +/-   ##
============================================
+ Coverage     92.14%   92.54%   +0.40%     
+ Complexity     7713     6995     -718     
============================================
  Files           933      901      -32     
  Lines         30823    29525    -1298     
  Branches       3762     3723      -39     
============================================
- Hits          28402    27325    -1077     
+ Misses         1575     1436     -139     
+ Partials        846      764      -82     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[jascks]

LGTM

[jnels124]

In previous pr to disable postgres node exporter container, we left configuration that allocates I beleive 100m cpu. Can we remove that and add in specific resource constraints for postgres-util?

[xin-hedera]

In previous pr to disable postgres node exporter container, we left configuration that allocates I beleive 100m cpu. Can we remove that and add in specific resource constraints for postgres-util?

My recollection of when I enabled it manually, the container had no resources section at all. Perhaps we should double check and give a reasonable cpu limit. I’d prefer getting it in a separate PR still.

[edwin-greene]

Looks good

[jnels124]

LGTM

[xin-hedera]

In previous pr to disable postgres node exporter container, we left configuration that allocates I beleive 100m cpu. Can we remove that and add in specific resource constraints for postgres-util?

My recollection of when I enabled it manually, the container had no resources section at all. Perhaps we should double check and give a reasonable cpu limit. I’d prefer getting it in a separate PR still.

Ticket here: #9166

[sonarcloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud

[jascks]

LGTM

[steven-sheehy]

LGTM