Support tracing via the ObservabilityPolicy

[sjberman]

Problem: As a user, I want to be able to enable tracing for requests flowing through NGF to my backend applications, so I can understand and debug my requests.

Solution: Using the ObservabilityPolicy, an app dev can now enable and configure tracing for their Route(s). The policy will only be applied if the NginxProxy configuration containing the tracing collector endpoint has been defined and attached to the GatewayClass. The policy can be attached to one or more HTTP or GRPC Routes.

Updated span attributes from the NginxProxy to be applied at the location block context, otherwise they are overwritten.

Also added functional tests to ensure that the complete solution is working.

Testing: Add functional tests for tracing

Verify end to end tracing by creating a collector and sending trace data to it.

Test cases:

1. one policy attached to one route
2. one policy attached to multiple routes

Also did lots of manual testing

Closes #1828
Closes #1911
Closes #1788

Checklist

Before creating a PR, run through this checklist and mark each as complete.

• I have read the CONTRIBUTING doc
• I have added tests that prove my fix is effective or that my feature works
• I have checked that all unit tests pass after adding my changes
• I have updated necessary documentation
• I have rebased my branch onto main
• I will ensure my PR is targeting the main branch and pulling from my branch from my own fork

Release notes

If this PR introduces a change that affects users and needs to be mentioned in the release notes,
please add a brief note that summarizes the change.

Support tracing via the ObservabilityPolicy CRD.

[codecov]

Codecov Report

Attention: Patch coverage is 92.50936% with 20 lines in your changes are missing coverage. Please review.

Project coverage is 87.61%. Comparing base (da36700) to head (ac840c4).

Files	Patch %	Lines
internal/mode/static/policies/policy.go	0.00%	16 Missing ⚠️
internal/mode/static/manager.go	20.00%	4 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2004      +/-   ##
==========================================
- Coverage   87.66%   87.61%   -0.05%     
==========================================
  Files          93       96       +3     
  Lines        6557     6694     +137     
  Branches       50       50              
==========================================
+ Hits         5748     5865     +117     
- Misses        753      773      +20     
  Partials       56       56              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[kate-osborn]

@sjberman can you add some tests here:

nginx-gateway-fabric/internal/mode/static/state/change_processor_test.go

Line 1665 in 35844ac

NOTE: When adding a new NGF policy to the change processor,

We just need to make sure the shared policy store correctly handles multiple policy types.

[kate-osborn]

🚀 🚀