Gateway and Virtual Service both failing to report missing secrets

[day0ops]

Gloo Edge Product

Enterprise

Gloo Edge Version

1.16.4

Kubernetes Version

v1.29.8

Describe the bug

While testing header secrets in Vault I came across what appears to be a translation bug but the bug appears to be affecting both Kubernetes and Vault storage so far. Its quite possible this bug does not affect other secret types but I have not tested others.

As below both Gateway and Virtual Service reporting as Accepted.

apiVersion: gateway.solo.io/v1
kind: Gateway
metadata:
  labels:
    app: gloo
  name: gateway-proxy
  namespace: gloo-system
spec:
  bindAddress: '::'
  bindPort: 8080
  httpGateway: {}
  proxyNames:
  - gateway-proxy
  ssl: false
  useProxyProto: false
status:
  statuses:
    gloo-system:
      reportedBy: gloo
      state: Accepted
      subresourceStatuses:
        '*v1.Proxy.gateway-proxy_gloo-system':
          reportedBy: gloo
          state: Accepted

and

apiVersion: gateway.solo.io/v1
kind: VirtualService
metadata:
  name: main-vs
  namespace: apps-configuration
spec:
  virtualHost:
    domains:
    - '*'
    routes:
    - matchers:
      - prefix: /
      options:
        headerManipulation:
          requestHeadersToAdd:
          - headerSecretRef:
              name: test-header
              namespace: apps-configuration
      routeAction:
        single:
          upstream:
            name: httpbin-upstream
            namespace: apps
status:
  statuses:
    gloo-system:
      reportedBy: gloo
      state: Accepted
      subresourceStatuses:
        '*v1.Proxy.gateway-proxy_gloo-system':
          reportedBy: gloo
          state: Accepted

This secret test-header does not exist.

❯ kubectl get secret -n apps-configuration
No resources found in apps-configuration namespace.

Controller logs showing the following error,

{"level":"warn","ts":"2024-10-17T04:23:13.451Z","logger":"gloo-ee.v1.event_loop.setup.gloosnapshot.event_loop.envoyTranslatorSyncer","caller":"syncer/envoy_translator_syncer.go:142","msg":"Proxy had invalid config","version":"1.16.4","proxy":"name:\"gateway-proxy\"  namespace:\"gloo-system\"","error":"2 errors occurred:\n\t* invalid resource gloo-system.gateway-proxy\n\t* Route Error: ProcessingError. Reason: *headers.plugin: list did not find secret apps-configuration.test-header. Route Name: apps-configuration_main-vs-route-0-matcher-0\n\n"}
{"level":"warn","ts":"2024-10-17T04:23:13.451Z","logger":"gloo-ee.v1.event_loop.setup.gloosnapshot.event_loop.envoyTranslatorSyncer","caller":"syncer/envoy_translator_syncer.go:150","msg":"Proxy had invalid config after xds sanitization","version":"1.16.4","proxy":"name:\"gateway-proxy\"  namespace:\"gloo-system\"","error":"2 errors occurred:\n\t* invalid resource gloo-system.gateway-proxy\n\t* Route Error: ProcessingError. Reason: *headers.plugin: list did not find secret apps-configuration.test-header. Route Name: apps-configuration_main-vs-route-0-matcher-0\n\n"}

Despite the missing secret the request is still processed successfully, but the new header is missing.

{
  "args": {},
  "headers": {
    "Accept": [
      "*/*"
    ],
    "Host": [
      "aae6fd8d791ff4f44a69aa8969b64048-1983600809.ap-southeast-1.elb.amazonaws.com"
    ],
    "User-Agent": [
      "curl/8.7.1"
    ],
    "X-Envoy-Expected-Rq-Timeout-Ms": [
      "15000"
    ],
    "X-Forwarded-Proto": [
      "http"
    ],
    "X-Request-Id": [
      "6486bc0d-1eeb-4321-b781-79c53da1a2df"
    ]
  },
  "origin": "192.168.84.182:42940",
  "url": "http://aae6fd8d791ff4f44a69aa8969b64048-1983600809.ap-southeast-1.elb.amazonaws.com/get"
}

Expected Behavior

Gateway and Virtual Service should report the missing secret.

Steps to reproduce the bug

Reproducer as given below.
https://github.com/day0ops/solo-reproducers/tree/main/gloo-10210

Additional Environment Detail

Tested on both 1.16.4 and 1.17.3.

Client Version: v1.31.1
Kustomize Version: v5.4.2
Server Version: v1.29.8-eks-a737599

Additional Context

Full logs from the controller
gloo.log