[Security Assistant] Adds client hooks and internal routes for managing Knowledge Base Entries

[spong]

Summary

This PR adds client hooks and basic REST API's for accessing and mutating Knowledge Base Entries. This is in support of @angorayc building out the new Knowledge Base settings interface.

Change set includes:

• Refactors existing KB client hooks from x-pack/packages/kbn-elastic-assistant/impl/knowledge_base to be co-located next to the API methods where we put all our other hooks: x-pack/packages/kbn-elastic-assistant/impl/assistant/api/knowledge_base
• Refactors existing KB API calls and associated tests out of kbn-elastic-assistant/impl/assistant/api/index.tsx and into x-pack/packages/kbn-elastic-assistant/impl/assistant/api/knowledge_base/api.tsx
• Adds new find_knowledge_base_entries_route.schema.yaml OAS for the supporting /internal/elastic_assistant/knowledge_base/entries/_find route
• Refactors SortOrder out of existing OAS's into the shared schemas/common_attributes.schema.yaml

Client Hooks & Routes

Adds new useKnowledgeBaseEntries() hook and corresponding /knowledge_base/entries/_find route for returning paginated KB Entries to populate the KB table in settings. E.g.

    const {
      assistantFeatures: { assistantKnowledgeBaseByDefault: enableKnowledgeBaseByDefault },
      http,
      toasts,
    } = useAssistantContext();
    const { data: kbEntries, isLoading: isLoadingEntries } = useKnowledgeBaseEntries({ http });

Sample Response

{
  "perPage": 20,
  "page": 1,
  "total": 145,
  "data": [
    {
      "timestamp": "2024-06-05T21:19:56.482Z",
      "id": "CtBF6o8BSQy1Bdxt2FHz",
      "createdAt": "2024-06-05T21:19:56.482Z",
      "createdBy": "u_mGBROF_q5bmFCATbLXAcCwKa0k8JvONAwSruelyKA5E_0",
      "updatedAt": "2024-06-05T21:19:56.482Z",
      "updatedBy": "u_mGBROF_q5bmFCATbLXAcCwKa0k8JvONAwSruelyKA5E_0",
      "users": [
        {
          "id": "u_mGBROF_q5bmFCATbLXAcCwKa0k8JvONAwSruelyKA5E_0",
          "name": "elastic"
        }
      ],
      "metadata": {
        "kbResource": "security_labs",
        "source": "/Users/garrettspong/dev/kibana-main/x-pack/plugins/elastic_assistant/server/knowledge_base/security_labs/2022_elastic_global_threat_report_announcement.mdx",
        "required": false
      },
      "namespace": "default",
      "text": "[Source Content Here]",
      "vector": {
        "modelId": ".elser_model_2",
        "tokens": {
          "2": 0.06595266,
          ...
        }
      }
    },
    ...
  ]
}

Response is the full newly created entry. Same format for the entry as above in the _find API, and the KnowledgeBaseEntries cache is invalidated.

Adds new useCreateKnowledgeBaseEntry() hook and corresponding /knowledge_base/entries route for creating new KB Entries

    const entry: KnowledgeBaseEntryCreateProps = {
      metadata: {
        kbResource: 'user',
        required: true,
        source: 'user',
      },
      text: 'Useful information about the user',
    };
    const { mutate: createEntry, isLoading: isCreatingEntry } = useCreateKnowledgeBaseEntry({
      http,
    });
    await createEntry(entry);

Adds new useDeleteKnowledgeBaseEntries() hook and corresponding /knowledge_base/entries/_bulk_action route for deleting existing KB Entries. I left a TODO to plumb through delete_by_query so we can add a filter bar to the table. Need to confirm if we can do pagination with similarity search as well.

    const { mutate: deleteEntries, isLoading: isDeletingEntries } = useDeleteKnowledgeBaseEntries({
      http,
    });
    await deleteEntries({ ids: ['YOE_CZABSQy1BdxtAGbs'] })

See KnowledgeBaseEntryBulkCrudActionResponse for response formats. KnowledgeBaseEntries cache is invalidated upon delete.

Checklist

Delete any items that are not applicable to this PR.

• Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support
• Documentation was added for features that require explanation or tutorials
  • Feature currently behind feature flag. Documentation to be added before flag is removed. Tracked in [Request] 8.16 Security Assistant Custom Knowledge Bases security-docs#5337
• Unit or functional tests were updated or added to match the most common scenarios
  • API tests will need to be rounded out as we finalize functionality behind the feature flag

[spong]

Moving to internal routes and actually registering them. Routes contain logic for license/authUser/featureFlagEnabled checks.

[spong]

SortOrder was common across kb, conversations, and anonymization, so extracted to common attributes.

[spong]

I would like to support semantic search in this API as well (already plumbed it out in kbDataClient over in the sec-labs-kb PR). Will probably expose something like search_type:semantic param to use it, and then re-use filter param. If there is a more standard pattern for supporting multiple search types in the same API I can use that.

Not sure how pagination and stable references work with semantic search, so should verify that behavior first.

[spong]

Noticed our routes are starting to bloat with common checks like license, authUser or if a feature flag is enabled, so I made this helper to do any or all of those checks in a single block.

[spong]

This route was not previously registered, but it is now as of this PR, so marking as internal until we're ready.

[spong]

Will be removing castings over in the KB content PR (as loaders become generalized): #184885

[spong]

@elasticmachine merge upstream

[kibanamachine]

Flaky Test Runner Stats

🟠 Some tests failed. - kibana-flaky-test-suite-runner#6327

[❌] x-pack/test/security_solution_api_integration/test_suites/genai/nlp_cleanup_task/basic_license_essentials_tier/configs/serverless.config.ts: 20/25 tests passed.

see run history

[andrew-goldstein]

Consider allowing errors to propagate as an alternative to returning IHttpFetchError in this function, and the others in this file

[spong]

This was one of the existing hooks that got moved, so will live as-is for the time being. The new hooks propagate errors and return default data so are easier to work with. Will update these as the the cleanup work around the KB continues.

[andrew-goldstein]

Thanks @spong for the new KB hooks and routes! 🙏
LGTM 🚀

[spong]

@elasticmachine merge upstream

[angorayc]

Thanks for creating the knowledge base routes @spong !

[angorayc]

In this case we only retrieve 100 knowledge base maximum and no pagination, right?

[spong]

Yeah, this is leftover from using the conversations API as a template. I will be sure to plumb through pagination to the hook here 👍

[angorayc]

Suggested change

	* @param {Object} options - The options object.
	* @param {HttpSetup} options.http - HttpSetup
	* @param {Function} [options.onFetch] - transformation function for kb entries fetch result
	* @param {AbortSignal} [options.signal] - AbortSignal
	* @param {HttpSetup} options.http - HttpSetup
	* @param {AbortSignal} [options.signal] - AbortSignal

[angorayc]

@elasticmachine merge upstream

[kibana-ci]

💛 Build succeeded, but was flaky

• Buildkite Build
• Commit: 5478087

Failed CI Steps

• x-pack/test/alerting_api_integration/security_and_spaces/group1/config.ts
• FTR Configs #63

Test Failures

• [job] [logs] x-pack/test/alerting_api_integration/security_and_spaces/group1/config.ts / alerting api integration security and spaces enabled Alerts - Group 1 alerts backfill rule runs find backfill superuser at space1 should handle finding backfill requests with query string appropriately
• [job] [logs] FTR Configs #63 / discover/esql discover esql columns should reset columns only if index pattern changes in non-transformational query

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
securitySolution	5517	5519	+2

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
@kbn/elastic-assistant-common	276	286	+10

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
securitySolution	13.6MB	13.6MB	+2.4KB

Unknown metric groups

API count

id	before	after	diff
@kbn/elastic-assistant-common	295	305	+10

ESLint disabled line counts

id	before	after	diff
elasticAssistant	39	40	+1

Total ESLint disabled count

id	before	after	diff
elasticAssistant	39	40	+1

History

• 💛 Build #216214 was flaky 5a62842
• 💔 Build #216019 failed 8645635
• 💛 Build #215950 was flaky 24c5701
• 💔 Build #215948 failed 0ac5d0f
• 💚 Build #215936 succeeded af3e139
• 💚 Build #215375 succeeded 6c36192

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @spong