Add optional description parameter to ingest processors.

[talevy]

This commit adds an optional field, description, to all ingest processors
so that users can explain the purpose of the specific processor instance.

Closes #56000.

[elasticmachine]

Pinging @elastic/es-core-features (:Core/Features/Ingest)

[talevy]

I still need to do a BWC check. but it is not clear to me how to do this best. simply create a pipeline with processors with a description in a mixed-cluster and then read? and hope via randomization that we get an old node reading the pipeline?

[jakelandis]

I still need to do a BWC check.

I don't think so... the only BWC concern with the changes here are on serialization of the JSON for response to the GET pipeline REST action and read/write from cluster state. The former just delegates to XContentHelper.convertToMap and the latter is stored via BytesReference and gets inflated to a Map. I don't think there is anything specific to the changes here w.r.t BWC that need to be tested.

FWIW, I manually tested writing the new description field to cluster state with the code here, and read it back in with master (that does not contain the code here). Because of the general serialization BytesReference and Maps, the description actually displays on the earlier versions that don't support description.

[jakelandis]

Did you want to include support for simulate?verbose on this PR ? If not, I have another inflight PR that touches the simulate stuff i can address there.

[jakelandis]

LGTM - as is.

However a minor addition to the 20_crud.yml test would be nice.

[talevy]

thanks @jakelandis! getting to this all now

[talevy]

Did you want to include support for simulate?verbose on this PR ?

I did not think it would be of value to include the description in each verbose processor response. @jloleysens, is this something that would be needed by the the pipeline builder?

[talevy]

fyi, the behavior is that a mixed cluster will not allow pipelines with description in the processors

{
  "error" : {
    "root_cause" : [
      {
        "type" : "parse_exception",
        "reason" : "processor [set] doesn't support one or more provided configuration parameters [description]",
        "processor_type" : "set"
      }
    ],
    "type" : "parse_exception",
    "reason" : "processor [set] doesn't support one or more provided configuration parameters [description]",
    "processor_type" : "set"
  },
  "status" : 400
}

[jloleysens]

I did not think it would be of value to include the description in each verbose processor response. @jloleysens, is this something that would be needed by the the pipeline builder?

@talevy

Yeah I'm not sure that this is desirable. With the work being done on the _simulate endpoint I think we will be able to associate a verbose result with a specific processor - and so still have access to the that information for a given response. The tying back from a result to the processor instance, user side, is the important bit.

[talevy]

thanks @jloleysens, that was helpful!

[talevy]

@elasticmachine run elasticsearch-ci/bwc