Add possibility to provide custom pod annotations (#2250)

Problem: Currently the helm chart does not provide an option to 
configure custom annotations for the nginx-gateway-fabric pod(s).
In our scenario, this is inconvenient because the pre-configured annotations 
Prometheus specific, but we are using Grafana Alloy to scrape metrics and Alloy 
is expecting different annotations. Of course there are also other scenarios where 
custom annotations can be useful, for instance to configure exceptions for Gatekeeper policies.

Solution: This Pull Request adds the possibility to configure your own 
custom annotations which will be added to the nginx-gateway-fabric pods.

charts/nginx-gateway-fabric/README.md

@@ -285,6 +285,7 @@ The following table lists the configurable parameters of the NGINX Gateway Fabri
 | `nginxGateway.leaderElection.enable` | Enable leader election. Leader election is used to avoid multiple replicas of the NGINX Gateway Fabric reporting the status of the Gateway API resources. If not enabled, all replicas of NGINX Gateway Fabric will update the statuses of the Gateway API resources. | bool | `true` |
 | `nginxGateway.leaderElection.lockName` | The name of the leader election lock. A Lease object with this name will be created in the same Namespace as the controller. | string | Autogenerated if not set or set to "". |
 | `nginxGateway.lifecycle` | The lifecycle of the nginx-gateway container. | object | `{}` |
+| `nginxGateway.podAnnotations` | Set of custom annotations for the NGINX Gateway Fabric pods. | object | `{}` |
 | `nginxGateway.productTelemetry.enable` | Enable the collection of product telemetry. | bool | `true` |
 | `nginxGateway.readinessProbe.enable` | Enable the /readyz endpoint on the control plane. | bool | `true` |
 | `nginxGateway.readinessProbe.initialDelaySeconds` | The number of seconds after the Pod has started before the readiness probes are initiated. | int | `3` |

charts/nginx-gateway-fabric/templates/deployment.yaml

@@ -15,13 +15,18 @@ spec:
     metadata:
       labels:
       {{- include "nginx-gateway.selectorLabels" . | nindent 8 }}
-      {{- if .Values.metrics.enable }}
+      {{- if or .Values.nginxGateway.podAnnotations .Values.metrics.enable }}
       annotations:
+        {{- if .Values.nginxGateway.podAnnotations }}
+        {{- toYaml .Values.nginxGateway.podAnnotations | nindent 8 }}
+        {{- end }}
+        {{- if .Values.metrics.enable }}
         prometheus.io/scrape: "true"
         prometheus.io/port: "{{ .Values.metrics.port }}"
         {{- if .Values.metrics.secure }}
         prometheus.io/scheme: "https"
         {{- end }}
+        {{- end }}
       {{- end }}
     spec:
       containers:

charts/nginx-gateway-fabric/values.yaml

@@ -7,6 +7,9 @@ nginxGateway:
   # belong to its class - i.e. have the "gatewayClassName" field resource equal to the class.
   gatewayClassName: nginx
 
+  # -- Set of custom annotations for the NGINX Gateway Fabric pods.
+  podAnnotations: {}
+
   # -- Set of custom annotations for GatewayClass objects.
   gatewayClassAnnotations: {}