Clarify usage of otelcol.processor.discovery (grafana#5378)

* Clarify usage of otelcol.processor.discovery

Co-authored-by: Clayton Cornell <131809008+clayton-cornell@users.noreply.github.com>

component/otelcol/processor/discovery/discovery_test.go

@@ -145,7 +145,8 @@ func Test_Insert(t *testing.T) {
 		targets = [{
 			"__address__" = "1.2.2.2", 
 			"__internal_label__" = "test_val",
-			"test_label" = "test_val2"}]
+			"test_label" = "test_val2",
+			"test.label.with.dots" = "test.val2.with.dots"}]
 
 		operation_type = "insert"
 
@@ -169,6 +170,10 @@ func Test_Insert(t *testing.T) {
 				{
 					"key": "test_label",
 					"value": { "stringValue": "old_val" }
+				},
+				{
+					"key": "test.label.with.dots",
+					"value": { "stringValue": "old_val" }
 				}]
 			},
 			"scopeSpans": [{
@@ -210,6 +215,10 @@ func Test_Insert(t *testing.T) {
 				{
 					"key": "test_label",
 					"value": { "stringValue": "old_val" }
+				},
+				{
+					"key": "test.label.with.dots",
+					"value": { "stringValue": "old_val" }
 				}]
 			},
 			"scopeSpans": [{
@@ -231,6 +240,10 @@ func Test_Insert(t *testing.T) {
 				{
 					"key": "test_label",
 					"value": { "stringValue": "test_val2" }
+				},
+				{
+					"key": "test.label.with.dots",
+					"value": { "stringValue": "test.val2.with.dots" }
 				}]
 			},
 			"scopeSpans": [{
@@ -253,7 +266,8 @@ func Test_Update(t *testing.T) {
 		targets = [{
 			"__address__" = "1.2.2.2", 
 			"__internal_label__" = "test_val",
-			"test_label" = "test_val2"}]
+			"test_label" = "test_val2",
+			"test.label.with.dots" = "test.val2.with.dots"}]
 
 		operation_type = "update"
 
@@ -277,6 +291,10 @@ func Test_Update(t *testing.T) {
 				{
 					"key": "test_label",
 					"value": { "stringValue": "old_val" }
+				},
+				{
+					"key": "test.label.with.dots",
+					"value": { "stringValue": "old_val" }
 				}]
 			},
 			"scopeSpans": [{
@@ -318,6 +336,10 @@ func Test_Update(t *testing.T) {
 				{
 					"key": "test_label",
 					"value": { "stringValue": "test_val2" }
+				},
+				{
+					"key": "test.label.with.dots",
+					"value": { "stringValue": "test.val2.with.dots" }
 				}]
 			},
 			"scopeSpans": [{

docs/sources/flow/reference/components/otelcol.processor.discovery.md

@@ -16,12 +16,38 @@ of labels for each discovered target.
 `otelcol.processor.discovery` adds resource attributes to spans which have a hostname 
 matching the one in the `__address__` label provided by the `discovery.*` component.
 
-> **NOTE**: `otelcol.processor.discovery` is a custom component unrelated to any
-> processors from the OpenTelemetry Collector.
+{{% admonition type="note" %}}
+`otelcol.processor.discovery` is a custom component unrelated to any
+processors from the OpenTelemetry Collector.
+{{% /admonition %}}
 
 Multiple `otelcol.processor.discovery` components can be specified by giving them
 different labels.
 
+{{% admonition type="note" %}}
+It can be difficult to follow [OpenTelemetry semantic conventions][OTEL sem conv] when 
+adding resource attributes via `otelcol.processor.discovery`:
+* `discovery.relabel` and most `discovery.*` processes such as `discovery.kubernetes` 
+  can only emit [Prometheus-compatible labels][Prometheus data model].
+* Prometheus labels use underscores (`_`) in labels names, whereas 
+  [OpenTelemetry semantic conventions][OTEL sem conv] use dots (`.`).
+* Although `otelcol.processor.discovery` is able to work with non-Prometheus labels
+  such as ones containing dots, the fact that `discovery.*` components are generally 
+  only compatible with Prometheus naming conventions makes it hard to follow OpenTelemetry 
+  semantic conventions in `otelcol.processor.discovery`.
+
+If your use case is to add resource attributes which contain Kubernetes metadata, 
+consider using `otelcol.processor.k8sattributes` instead.
+
+------
+The main use case for `otelcol.processor.discovery` is for users who migrate to Grafana Agent Flow mode
+from Static mode's `prom_sd_operation_type`/`prom_sd_pod_associations` [configuration options][Traces].
+
+[Prometheus data model]: https://prometheus.io/docs/concepts/data_model/#metric-names-and-labels
+[OTEL sem conv]: https://github.com/open-telemetry/semantic-conventions/blob/main/docs/README.md
+[Traces]: {{< relref "../../../static/configuration/traces-config.md" >}}
+{{% /admonition %}}
+
 ## Usage
 
 ```river
@@ -146,16 +172,18 @@ otelcol.processor.discovery "default" {
 
 ### Using a preconfigured list of attributes
 
-It is not necessary to use a discovery component. In the example below, a `test_label` resource 
-attribute will be added to a span if its IP address is "1.2.2.2". The `__internal_label__` will
-be not be added to the span, because it begins with a double underscore (`__`).
+It is not necessary to use a discovery component. In the example below, both a `test_label` and 
+a `test.label.with.dots` resource attributes will be added to a span if its IP address is 
+"1.2.2.2". The `__internal_label__` will be not be added to the span, because it begins with 
+a double underscore (`__`).
 
 ```river
 otelcol.processor.discovery "default" {
     targets = [{
-        "__address__"        = "1.2.2.2", 
-        "__internal_label__" = "test_val",
-        "test_label"         = "test_val2"}]
+        "__address__"          = "1.2.2.2", 
+        "__internal_label__"   = "test_val",
+        "test_label"           = "test_val2",
+        "test.label.with.dots" = "test.val2.with.dots"}]
 
     output {
         traces = [otelcol.exporter.otlp.default.input]