[Ingest Pipelines][Append Processor] Adding a "unique" argument

[P1llus]

When appending values to specific fields there is currently a need to check if the related value already exists with if conditions.

It would be great to add support to check if the value already exists in the array and only append it if it does not.

In both Observability and Security usecases, we have the possibility to append related sources to make visualization easier. Fields like "related.user", "related.hash", "related.XYZ" is examples on that.

The issue comes down to certain cases where maybe the related user would be in multiple parts of the event.

POST /_ingest/pipeline/_simulate
{
  "pipeline": {
    "description": "_description",
    "processors": [
      {
        "append": {
          "field": "related.user",
          "value": "{{source.user}}"
        }
      },
      {
        "append": {
          "field": "related.user",
          "value": "{{client.user}}"
        }
      }
    ]
  },
  "docs": [
    {
      "_index": "index",
      "_id": "id",
      "_source": {
        "source": {
          "user": "testuser"
        },
        "client": {
          "user": "testuser"
        }
      }
    }
  ]
}

If we could add a "unique: true" argument to the append processor here, we could do something like:

{
  "append": {
    "field": "related.user",
    "value": "{{client.user}}",
    "unique": true
  }
}

[elasticmachine]

Pinging @elastic/es-core-features (:Core/Features/Ingest)