feat: immediate limits sync to pod via annotation

[KevFan]

Description

Part of Kuadrant/kuadrant-operator#414

This PR adds a feature to annotate Limitador pods with the limits config map resource version. This change ensures that modifications to the limits will promptly trigger a reload or synchronization of the limit configuration to Limitador pods. Consequently, it ensures that pod limits are always immediately up to date.

This enhancement will enable us to utilize the Limitador CR Ready condition as a way of naively assuming that limits are enforced.

Other additions include

• Updating kind to v0.22.0
• Refactor integration tests to use context to timeout tests and algin primarily use Gomega for assertions

Verification

The following have already been verified in the included integration test but if you want to verify manually:

• Checkout this branch
• Create local cluster

 make local-setup

• Watch for limitador pod annotations

watch "kubectl get pods -o yaml | yq '.items[].metadata.annotations'"

• Deploy limitador CR

kubectl apply -f -<<EOF                                          
apiVersion: limitador.kuadrant.io/v1alpha1
kind: Limitador
metadata:
  name: limitador-sample
spec:
  replicas: 2
  limits:
    - conditions: ["get_toy == 'yes'"]
      max_value: 10
      namespace: toystore-app
      seconds: 30
      variables: []
EOF

• Verify annotation is added to the pods
• Watch limits in a limitador pod

watch "kubectl exec $(kubectl get pods -l app=limitador -o yaml | yq '.items[0].metadata.name') -c limitador -- cat /home/limitador/etc/limitador-config.yaml"

• Update limits

kubectl apply -f -<<EOF                                          
apiVersion: limitador.kuadrant.io/v1alpha1
kind: Limitador
metadata:
  name: limitador-sample
spec:
  replicas: 2
  limits:
    - conditions: ["get_toy == 'yes'"]
      max_value: 20
      namespace: toystore-app
      seconds: 60
      variables: []
EOF

• Verify annotation of config map resource version is updated
• Verify config map in pod is immediately updated / synced to new value once annotation is updated

[codecov-commenter]

Codecov Report

Merging #127 (f0b39d5) into main (e9207fb) will increase coverage by 0.14%.
The diff coverage is 60.97%.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #127      +/-   ##
==========================================
+ Coverage   68.19%   68.34%   +0.14%     
==========================================
  Files          16       16              
  Lines        1154     1191      +37     
==========================================
+ Hits          787      814      +27     
- Misses        325      334       +9     
- Partials       42       43       +1     

Flag	Coverage Δ
integration	73.69% <68.57%> (+1.12%)	⬆️
unit	65.79% <16.66%> (-0.50%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
api/v1alpha1 (u)	90.00% <0.00%> (-10.00%)	⬇️
pkg/helpers (u)	∅ <ø> (∅)
pkg/log (u)	31.81% <ø> (ø)
pkg/reconcilers (u)	36.76% <ø> (ø)
pkg/limitador (u)	88.06% <100.00%> (-0.07%)	⬇️
controllers (i)	73.69% <68.57%> (+1.12%)	⬆️

Files	Coverage Δ
pkg/limitador/k8s_objects.go	80.18% <100.00%> (-0.19%)	⬇️
api/v1alpha1/limitador_types.go	90.00% <0.00%> (-10.00%)	⬇️
controllers/limitador_controller.go	70.07% <68.57%> (+2.07%)	⬆️

[eguzki]

Verification steps working like a charm

nice that the steps create two replicas.

[eguzki]

Some minor comments and one "bigger" one I would like to discuss pros and cons.

[KevFan]

Resolved comments and rebased with main

[eguzki]

LGTM and the verification steps work with the resource version of the configmap