Fix otel-collector logging repeat error

The pod log shows repeat log of duplicate timeseries error that takes up logging quota. Although the metris are correctly being sent, the error message can be confusing to users.

The removal of `type` tag is causing one timeseries being exported multiple times in batch with and without the tag.

This change uses aggregation in metricstransfrom processor to eliminate the `type` tag from Monarch pipeline. Format follows otel-collector-contrib 0.54.0.

Tested locally e2e.

b/290678742

pkg/metrics/otel.go

@@ -163,10 +163,15 @@ processors:
       # These labels are useful to users, but too noisy for global aggregation.
       - key: commit
         action: delete
-      - key: type
-        action: delete
   metricstransform/kubernetes:
     transforms:
+      - include: api_duration_seconds
+        action: update
+        operations:
+          # Eliminate 'type' tag from metric using aggregation
+          - action: aggregate_labels
+            label_set: [status, operation]
+            aggregation_type: sum
       - include: declared_resources
         action: update
         new_name: current_declared_resources
@@ -179,6 +184,11 @@ processors:
       - include: apply_operations_total
         action: update
         new_name: apply_operations_count
+        operations:
+          # Eliminate 'type' tag from metric using aggregation
+          - action: aggregate_labels
+            label_set: [controller, operation, status]
+            aggregation_type: sum
       - include: resource_fights_total
         action: update
         new_name: resource_fights_count
@@ -191,6 +201,13 @@ processors:
       - include: rendering_count_total
         action: update
         new_name: rendering_count
+      - include: remediate_duration_seconds
+        action: update
+        operations:
+          # Eliminate 'type' tag from metric using aggregation
+          - action: aggregate_labels
+            label_set: [status]
+            aggregation_type: sum
       - include: skip_rendering_count_total
         action: update
         new_name: skip_rendering_count

pkg/reconcilermanager/controllers/otel_controller_test.go

@@ -46,7 +46,7 @@ const (
 	// otel-collector ConfigMap.
 	// See `CollectorConfigGooglecloud` in `pkg/metrics/otel.go`
 	// Used by TestOtelReconcilerGooglecloud.
-	depAnnotationGooglecloud = "00a85865e19d827bad96615656b05cef"
+	depAnnotationGooglecloud = "a489312e2ea53ecd5da8d4784feeaf2c"
 	// depAnnotationGooglecloud is the expected hash of the custom
 	// otel-collector ConfigMap test artifact.
 	// Used by TestOtelReconcilerCustom.