[Security Assistant] Updates Knowledge Base ES Schema, OAS, and API to support IndexEntries

[spong]

Summary

This is a follow-up to #184974 that updates the KB Entries API's (and underlying schema) to support IndexEntries in addition to DocumentEntries. IndexEntries are entries in the Knowledge Base that are not backed by an embedded raw text source, but rather by an entire Index or Data Stream. The user can set the data source name, the specific field to query (must be ELSER embeddings in this initial implementation), and a description for when the assistant should search this data source for Knowledge Base content. This essentially enables the user to create custom retrieval tools backed by their own data.

The changes in this PR, as with the other recent KB enhancements, are behind the following feature flag:

xpack.securitySolution.enableExperimental:
  - 'assistantKnowledgeBaseByDefault'

however as code change is required to test the new mappings. For this you can update the knowledgeBaseDataStream in x-pack/plugins/elastic_assistant/server/ai_assistant_service/index.ts to

    this.knowledgeBaseDataStream = this.createDataStream({
      resource: 'knowledgeBase',
      kibanaVersion: options.kibanaVersion,
      fieldMap: knowledgeBaseFieldMapV2, // Update this to the V2 mapping
    });

Change set includes:

• ES Knowledge Base data stream schema and OAS has been updated to support IndexEntries.
• OAS schema files have been moved to the /entries sub-directory
• Backend KB services have been updated to support IndexEntries
  • Storage methods updated
  • Retrieval methods updated (will round out these endpoint when working the UI next)

---

With these API changes, I've also introduced a few sample *.http files for easier development/testing. These files are supported out of the box in JetBrains IDE's or in VSCode with the httpyac (and many other) extensions. Since the configuration for these files includes a - in the name, that's why you'll see a few @elastic/kibana-operations files updated.

You can read more about http files here and for the spec see this repo here. If we find these useful, we could add support to our OpenAPI Generator to create these automatically. They currently live co-located next to the OAS and generated schema files here:

x-pack/packages/kbn-elastic-assistant-common/impl/schemas/knowledge_base/entries/bulk_crud_knowledge_base_entries_route.http
x-pack/packages/kbn-elastic-assistant-common/impl/schemas/knowledge_base/entries/crud_knowledge_base_entries_route.http

and the main config here:

x-pack/packages/kbn-elastic-assistant-common/env/http-client.env.json

The x-pack/packages/kbn-elastic-assistant-common/.gitignore has been updated to ignore http-client.private.env.json files locally, which is how you can override the config as you'd like. This is helpful to add variables like basePath as below:

{
  "dev": {
    "basePath": "/kbn"
  }
}

To use them, just open the corresponding *.http for the API you want to test, and click Send, and the response will open in another tab. Here is what that looks like for creating one of the new IndexEntry KB documents that have been introduced in this PR:

Checklist

Delete any items that are not applicable to this PR.

• Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support
• Documentation was added for features that require explanation or tutorials
  • Feature currently behind feature flag. Documentation to be added before flag is removed. Tracked in [Request] 8.16 Security Assistant Custom Knowledge Bases security-docs#5337
• Unit or functional tests were updated or added to match the most common scenarios

[andrew-goldstein]

No change requested, just noting that the ES mappings for some of the fields that EsIndexEntry represents are optional, for example:

  created_by: {
    type: 'keyword',
    array: false,
    required: false,
  },

I'm wondering if some of the transforms from ES results, like:

createdBy: esKbEntry.created_by,

might result in instances of EsIndexEntry that could lead to runtime errors.

If EsIndexEntry are sent over the wire, these errors may be caught by (Zod) schema validation, but I'm wondering if they should also be optional, for example:

created_by?: string;

[spong]

Thank you for bringing this up! For this I had followed the pattern the other persistence implementations has set forth, but I think this is a good discussion item as much like your comment below, any data issue could result in runtime errors. Now the likely hood of these should be low since this would only happen by direct access via ES, and not the to-be-public API, but the more guards the merrier in terms of application stability.

If we make the ES mappings match the HTTP API interface (in terms of optionality), we get that type safety on retrieval, which is nice if the data is used outside of returning through the API. The only real downside I can think of now is there might be less flexibility in modifying the API as any API change is now a mapping version change and may require a rollover/etc, which we don't fully have ironed out at the moment...

Would you be okay if we discuss this as a team WRT in related to all the other persistence implementations we have and maybe we can make this change all at once once we can smoothly perform mapping upgrades?

[andrew-goldstein]

nit: No change requested, but when I followed the chain of function calls that use this function, i.e. x-pack/plugins/elastic_assistant/server/routes/knowledge_base/entries/find_route.ts it appears a single document with an unexpected esKbEntry.type may cause the retrieval of all other (matching) documents to fail.

Consider returning null and forcing the caller to deal with errors via null checks if it's useful for the caller to still get the other (valid) docs.

[spong]

Good call! I've added a TODO and will look at this when wiring up the UI and making the remainder of the API changes to support bulk operations.

[andrew-goldstein]

Thanks @spong for this feature! 🙏
✅ Desk tested locally via the Slack connector
LGTM 🚀

[kibana-ci]

💚 Build Succeeded

• Buildkite Build
• Commit: f4116b9

Metrics [docs]

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
@kbn/elastic-assistant-common	320	363	+43

Unknown metric groups

API count

id	before	after	diff
@kbn/elastic-assistant-common	346	390	+44

History

• 💛 Build #232926 was flaky 3d34832
• 💚 Build #232653 succeeded 9989998
• 💛 Build #232116 was flaky 435f8c1
• 💔 Build #232105 failed 07c1ba3
• 💚 Build #232050 succeeded a0de4d1
• 💛 Build #231846 was flaky e1bde2c

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @spong