[ML] Alerts as data integration for Anomaly Detection rule type

[darnautov]

Summary

Part of #165958

Replaces usage of the deprecated alertFactory with the new alerts client and adds alerts-as-data integration for Anomaly Detection alerting rule type.

Alert instances are stored in .alerts-ml.anomaly-detection.alerts-default index and extends the common AlertSchema.

Result mappings

{
".internal.alerts-ml.anomaly-detection.alerts-default-000001": {
  "mappings": {
    "dynamic": "false",
    "_meta": {
      "namespace": "default",
      "kibana": {
        "version": "8.11.0"
      },
      "managed": true
    },
    "properties": {
      "@timestamp": {
        "type": "date"
      },
      "event": {
        "properties": {
          "action": {
            "type": "keyword"
          },
          "kind": {
            "type": "keyword"
          }
        }
      },
      "kibana": {
        "properties": {
          "alert": {
            "properties": {
              "action_group": {
                "type": "keyword"
              },
              "anomaly_score": {
                "type": "double"
              },
              "anomaly_timestamp": {
                "type": "date"
              },
              "case_ids": {
                "type": "keyword"
              },
              "duration": {
                "properties": {
                  "us": {
                    "type": "long"
                  }
                }
              },
              "end": {
                "type": "date"
              },
              "flapping": {
                "type": "boolean"
              },
              "flapping_history": {
                "type": "boolean"
              },
              "instance": {
                "properties": {
                  "id": {
                    "type": "keyword"
                  }
                }
              },
              "is_interim": {
                "type": "boolean"
              },
              "job_id": {
                "type": "keyword"
              },
              "last_detected": {
                "type": "date"
              },
              "maintenance_window_ids": {
                "type": "keyword"
              },
              "reason": {
                "type": "keyword"
              },
              "rule": {
                "properties": {
                  "category": {
                    "type": "keyword"
                  },
                  "consumer": {
                    "type": "keyword"
                  },
                  "execution": {
                    "properties": {
                      "uuid": {
                        "type": "keyword"
                      }
                    }
                  },
                  "name": {
                    "type": "keyword"
                  },
                  "parameters": {
                    "type": "flattened",
                    "ignore_above": 4096
                  },
                  "producer": {
                    "type": "keyword"
                  },
                  "revision": {
                    "type": "long"
                  },
                  "rule_type_id": {
                    "type": "keyword"
                  },
                  "tags": {
                    "type": "keyword"
                  },
                  "uuid": {
                    "type": "keyword"
                  }
                }
              },
              "start": {
                "type": "date"
              },
              "status": {
                "type": "keyword"
              },
              "time_range": {
                "type": "date_range",
                "format": "epoch_millis||strict_date_optional_time"
              },
              "top_influencers": {
                "type": "nested",
                "dynamic": "false",
                "properties": {
                  "influencer_field_name": {
                    "type": "keyword"
                  },
                  "influencer_field_value": {
                    "type": "keyword"
                  },
                  "influencer_score": {
                    "type": "double"
                  },
                  "initial_influencer_score": {
                    "type": "double"
                  },
                  "is_interim": {
                    "type": "boolean"
                  },
                  "job_id": {
                    "type": "keyword"
                  },
                  "timestamp": {
                    "type": "date"
                  }
                }
              },
              "top_records": {
                "type": "nested",
                "dynamic": "false",
                "properties": {
                  "actual": {
                    "type": "double"
                  },
                  "by_field_name": {
                    "type": "keyword"
                  },
                  "by_field_value": {
                    "type": "keyword"
                  },
                  "detector_index": {
                    "type": "integer"
                  },
                  "field_name": {
                    "type": "keyword"
                  },
                  "function": {
                    "type": "keyword"
                  },
                  "initial_record_score": {
                    "type": "double"
                  },
                  "is_interim": {
                    "type": "boolean"
                  },
                  "job_id": {
                    "type": "keyword"
                  },
                  "over_field_name": {
                    "type": "keyword"
                  },
                  "over_field_value": {
                    "type": "keyword"
                  },
                  "partition_field_name": {
                    "type": "keyword"
                  },
                  "partition_field_value": {
                    "type": "keyword"
                  },
                  "record_score": {
                    "type": "double"
                  },
                  "timestamp": {
                    "type": "date"
                  },
                  "typical": {
                    "type": "double"
                  }
                }
              },
              "url": {
                "type": "keyword",
                "index": false,
                "ignore_above": 2048
              },
              "uuid": {
                "type": "keyword"
              },
              "workflow_status": {
                "type": "keyword"
              },
              "workflow_tags": {
                "type": "keyword"
              }
            }
          },
          "space_ids": {
            "type": "keyword"
          },
          "version": {
            "type": "version"
          }
        }
      },
      "tags": {
        "type": "keyword"
      }
    }
  }
}
}

Checklist

• Documentation was added for features that require explanation or tutorials
• Unit or functional tests were updated or added to match the most common scenarios

[darnautov]

@droberts195 could you please review the mappings for alert-as-data index? only additive changes will be allowed after release so I want to double-check it's correct.

[droberts195]

I left some comments on the details but I don't understand enough about how these alerts will be searched to say with certainty what the correct answers are. There are very complicated tradeoffs to be made. Whatever is done, it's important that some comments are added to say why we've decided on those tradeoffs, as it's inevitable that somebody will complain about our decision in the future whichever way we go.

The decision on the mappings here needs to go with a best practice guide on how to search the resulting alerts.

[droberts195]

nested is a very complex type to allow the sub-objects to be searched individually. It results in secret documents being indexed for each element of the array.

The other alternative here would be object. Whether nested is really needed here is probably the most important decision to make about these mappings before merging the PR. I don't know enough about how these alerts will be searched to know the answer.

There are more details in the "Arrays of objects" note on https://www.elastic.co/guide/en/elasticsearch/reference/current/array.html and the linked docs for the nested type.

The same question applies to influencers below.

[darnautov]

We discussed it with @peteharverson and agreed it might be useful to fetch alerts with certain influencers, e.g.

GET .alerts-ml.anomaly-detection.alerts-default/_search
{
  "query": {
    "nested": {
      "path": "kibana.alert.top_influencers",
      "query": {
        "bool": {
          "must": [
            {
              "match": {
                "kibana.alert.top_influencers.influencer_field_name": "key"
              }
            }
          ]
        }
      }
    }
  }
}

If I'm not mistaken it's only possible with the nested type.

[kobelb]

I agree with @droberts195 that we need to be especially cautious with using the nested type. There are tons of pitfalls with the nested field type per the official docs: https://www.elastic.co/guide/en/elasticsearch/reference/current/nested.html. Additionally, Kibana does not handle nested fields well, so this will make consuming the data stored in these nested field types complicated as well.

If we use the object type, we'd still be able to perform the query to match the influencers. The biggest limitation is that we can't query for multiple fields that appear within the array because of the way ES flattens arrays: https://www.elastic.co/guide/en/elasticsearch/reference/current/nested.html#nested-arrays-flattening-objects

[droberts195]

We're running into general limitations of Elasticsearch and Kibana functionality.

"Show me ML alerts where one of the influencer fields was airline" will be possible with an object mapping.

But, "Show me ML alerts where one of the influencer fields was airline and its influencer score was greater than 90" will not be reliable with object. This is because influence_field_name: airline and influencer_score > 90 might match on different array elements. So it might be that destination_airport was the influencer with the score above 90, not airline.

Given the difficulties of working with nested in Kibana, and other drawbacks with flattened, it sounds like object is the mapping to go for here. But this will restrict the search functionality that can be built on top of these alerts to some extent. If these limitations are likely to cause problems then we need to hold this functionality back for a later release and think through a more major redesign of the document structure.

[darnautov]

Many thanks for the feedback and suggestions @droberts195, @kobelb. Updated to the object type in 33d037c.

Support for queries like this should be sufficient, cc @peteharverson

GET .alerts-ml.anomaly-detection.alerts-default/_search
{
  "query": {
    "term": {
      "kibana.alert.top_influencers.influencer_field_name": {
        "value": "key"
      }
    }
  }
}

[droberts195]

Thanks Dima. We discussed this a bit and came to the conclusion that object was the best option of the 3, even though none of the options are perfect. So what you've done tallies with the conclusion we came to.

[droberts195]

One thing to understand here is that when we write anomalies to the anomaly results indices we added the partition/by/over fields twice:

1. We add them as partition_field_name/partition_field_value etc.
2. We add them using their original field names, for example airline/AAL

In the anomaly results indices we try to map the original field names as well as the predictable ones. This means that you can search anomaly results with a terms search on the airline field and get hits.

Setting dynamic: false here means that these sub-objects won't be searchable by original field names. If the values have been populated by copying the _source of the original anomalies in full then the original fields will be visible in the _source of the alerts, but the alerts won't be searchable by these original field names.

Unfortunately, trying to create the original field names in the mappings in the anomaly results indices has turned out to be a very difficult problem, and has resulted in a lot of complicated code and support cases when things go wrong.

Therefore I think it would be best not to try and do the same for alerts. (In other words leave this part of the mappings as it is now.) But it's important to realise that, as a result, it won't be possible to search for alerts for airline/AAL, but instead it will be necessary to search for partition_field_name/airline and partition_field_value/AAL. This then ties back to the object vs nested decision, because you can only search for two fields together in arrays of objects if you use the nested type.

[darnautov]

Good point regarding searching by the original field names and allowing dynamic mappings. It only affects .alerts-ml.anomaly-detection.alerts-default indices so it shouldn't be an issue.
To clarify, will we be able to query for airline:AAL with the object type?

[ymao1]

Would it be possible to use the flattened mapping to be able to search for airline:AAL inside the copy of the original _source?

[darnautov]

I think it's important to know the actual field name. With the flattened mapping we can end up with similar field values coming from completely different fields if I understand it correctly. WDYT @droberts195?

[droberts195]

To clarify, will we be able to query for airline:AAL with the object type?

No, only if there was an explicit mapping for airline within the object. And getting those explicit mappings in is extremely hard for fields that vary by job.

Would it be possible to use the flattened mapping

flattened might be ideal if alert_top_records was a single object rather than an array. I am not sure how flattened will behave with an array of objects though - that would need some experimentation.

One more thing about flattened is that it treats all sub-values as keywords. So then it definitely won't be possible to do a search like, "Give me all alerts where one of the top records has a record score greater than 90."

It all comes back to how we expect users to be searching these alerts. If we have a feel for how people will want to search, filter and sort then it will be easier to create mappings that support those use cases (or at least say with certainty that it's impossible to support all of them and document the use cases we do support).

[darnautov]

Updated type to object in 33d037c

[droberts195]

I left some comments on the details but I don't understand enough about how these alerts will be searched to say with certainty what the correct answers are. There are very complicated tradeoffs to be made. Whatever is done, it's important that some comments are added to say why we've decided on those tradeoffs, as it's inevitable that somebody will complain about our decision in the future whichever way we go.

The decision on the mappings here needs to go with a best practice guide on how to search the resulting alerts.

[ymao1]

@XavierM Does the alerts table support KQL queries against nested fields? I don't believe we have any other nested fields in the AAD indices as of now.

[droberts195]

The index mappings LGTM.

(I haven't reviewed the rest of the PR, only the index mappings.)

[darnautov]

@elasticmachine merge upstream

[jgowdyelastic]

Code LGTM

[kibana-ci]

💚 Build Succeeded

• Buildkite Build
• Commit: 58b1aa9

Metrics [docs]

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
@kbn/alerts-as-data-utils	29	30	+1

Public APIs missing exports

Total count of every type that is part of your API that should be exported but is not. This will cause broken links in the API documentation system. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats exports for more detailed information.

id	before	after	diff
ml	32	33	+1

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
alerting	19.8KB	19.8KB	+4.0B

Unknown metric groups

API count

id	before	after	diff
@kbn/alerts-as-data-utils	29	30	+1

References to deprecated APIs

id	before	after	diff
ml	151	149	-2

History

• 💔 Build #161893 failed 01ecd1a
• 💔 Build #161713 failed 33d037c
• 💚 Build #160299 succeeded ad4b287
• 💚 Build #160256 succeeded 72d616f
• 💚 Build #160031 succeeded 90a87ed

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @darnautov

[peteharverson]

LGTM.
Tested against some anomaly detection rules and inspected index mapping and docs.