[Security Solution][Endpoint] Update to Endpoint List and associated supporting API to be space aware #194312

paul-tavares · 2024-09-27T15:10:46Z

Summary

PR makes the following changes in support of Endpoint management support for Spaces:

Fleet

Adds new method - getByIds() - to the server-side Agent service
Updates some mocks

Security Solution

Updates the Endpoint host metadata API for both the list and single endpoint to be space aware
Updates the agent status API for Endpoint to be space aware
Updates endpoint policy response API to be space aware
New FTR API integration test suite was created
- included changes to some service utilities to enable loading test data per-space

Testing

Enable the following feature keys:
- xpack.securitySolution.enableExperimental.endpointManagementSpaceAwarenessEnabled
- xpack.fleet.enableExperimental.useSpaceAwareness
Trigger migration in fleet to be space aware: API POST /internal/fleet/enable_space_awareness
Create data in multiple spaces and validate that Endpoint lIst/details/policy response display only appropriate data

Tip

As an alternative to step 2 and 3 above, you can also just run the node x-pack/plugins/security_solution/scripts/endpoint/run_endpoint_agent.js script with the --spaceId value and it will take care of adding a running endpoint to the given space

Checklist

Unit or functional tests were updated or added to match the most common scenarios

…t endpoint context services

…int metadata to use it

paul-tavares · 2024-09-27T19:34:43Z

/ci

…int-list-api-support-spaces-2

paul-tavares · 2024-10-02T19:21:08Z

/ci

…int-list-api-support-spaces-2

paul-tavares · 2024-10-03T15:26:25Z

/ci

…es-2

paul-tavares · 2024-10-09T12:31:32Z

Flaky test run: https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/7112

juliaElastic

Fleet change LGTM

kibanamachine · 2024-10-09T13:12:29Z

Flaky Test Runner Stats

🟠 Some tests failed. - kibana-flaky-test-suite-runner#7112

[❌] x-pack/test/security_solution_api_integration/test_suites/edr_workflows/spaces/trial_license_complete_tier/configs/ess.config.ts: 3/25 tests passed.

see run history

…es-2

…eported by endpoint in the data indexed

paul-tavares · 2024-10-09T15:13:21Z

Second flaky run: https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/7113

kibanamachine · 2024-10-09T16:06:13Z

Flaky Test Runner Stats

🎉 All tests passed! - kibana-flaky-test-suite-runner#7113

[✅] x-pack/test/security_solution_api_integration/test_suites/edr_workflows/artifacts/trial_license_complete_tier/configs/ess.config.ts: 25/25 tests passed.

see run history

…es-2

ashokaditya

Thanks for the changes. I did a code review only. I've a few nits but it looks good to ship. I'll test locally and report back if I find anything odd.

ashokaditya · 2024-10-16T15:32:51Z

x-pack/plugins/fleet/server/services/agents/agent_service.ts

@@ -128,6 +139,14 @@ class AgentClientImpl implements AgentClient {
    return getAgentById(this.internalEsClient, this.soClient, agentId);
  }

+  public async getByIds(


Perhaps rename this to getAgentsByIds?

Thanks for the feedback @ashokaditya . I named getByIds() to mirror other APIs available from fleet - thus to remain consistent with their naming conventions.

ashokaditya · 2024-10-16T15:37:57Z

x-pack/plugins/security_solution/common/endpoint/data_loaders/utils.ts

+    .request<Space>({
+      method: 'GET',
+      path: `/internal/spaces/_active_space`,
+      headers: { 'elastic-api-version': '2023-10-31' },


I think we use 1 for version headers for internal APIs.

I'll remove the header - looks like this API has not been moved to the versioned router:

kibana/x-pack/plugins/spaces/server/routes/api/internal/get_active_space.ts

Lines 15 to 19 in fe0ffab

router.get(

{

path: '/internal/spaces/_active_space',

validate: false,

},

ashokaditya · 2024-10-16T15:42:05Z

x-pack/plugins/security_solution/common/endpoint/data_loaders/index_fleet_server.ts

      const lastCheckin = new Date();
+      let agentVersion = version;
+
+      if (isServerless) {


isServerless would always be false here as it's wrapped in a block that checks for !sServerless.

ashokaditya · 2024-10-16T15:51:24Z

...olution/server/endpoint/services/agent/clients/endpoint/endpoint_agent_status_client.test.ts

+    // FIXME:PT need to remove the need for this mock. It appears in several test files on our side.
+    //  Its currently needed due to the direct use of Fleet's `buildAgentStatusRuntimeField()` in
+    //  `x-pack/plugins/security_solution/server/endpoint/routes/metadata/query_builders.ts:239`


ashokaditya · 2024-10-16T15:52:43Z

...ity_solution/server/endpoint/services/agent/clients/endpoint/endpoint_agent_status_client.ts

@@ -16,8 +16,11 @@ export class EndpointAgentStatusClient extends AgentStatusClient {
  protected readonly agentType: ResponseActionAgentType = 'endpoint';

  async getAgentStatuses(agentIds: string[]): Promise<AgentStatusRecords> {
-    const metadataService = this.options.endpointService.getEndpointMetadataService();
+    const soClient = this.options.soClient;


nit: const { soClient, esClient } = this.options;

ashokaditya · 2024-10-16T15:55:35Z

x-pack/plugins/security_solution/server/endpoint/services/metadata/endpoint_metadata_service.ts

+    const agentIds = (data?.hits?.hits || [])
+      .map((hit) => hit._source?.agent.id ?? '')
+      .filter((id) => !!id);


nit: You can use reduce to iterate over only once on the array.

ashokaditya · 2024-10-16T16:41:36Z

x-pack/plugins/fleet/server/services/agents/crud.ts

+ *
+ * @throws AgentNotFoundError
+ */
+export const getByIds = async (


Also perhaps rename this to getAgentsByIds

same comment as before - just following Fleet conventions as other getByIds() already exist

…support-spaces-2' into task/olm-8538-endpoint-list-api-support-spaces-2

…int-list-api-support-spaces-2

…es-2

kibanamachine · 2024-10-17T14:25:02Z

Starting backport for target branches: 8.x

https://github.com/elastic/kibana/actions/runs/11386984275

elasticmachine · 2024-10-17T14:25:34Z

💚 Build Succeeded

Buildkite Build
Commit: 9ef0bea

Metrics [docs]

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
`fleet`	1292	1296	+4

Unknown metric groups

API count

id	before	after	diff
`fleet`	1415	1421	+6

History

💔 Build #243316 failed 3b741e6
💔 Build #243291 failed 081389b
💔 Build #243276 failed b9b7f54
💛 Build #243150 was flaky bfbb682
💔 Build #243087 failed 12f5b9e
💛 Build #240769 was flaky c9f8c7e

cc @paul-tavares

…supporting API to be space aware (elastic#194312) ## Summary PR makes the following changes in support of Endpoint management support for Spaces: ### Fleet - Adds new method - `getByIds()` - to the server-side Agent service - Updates some mocks ### Security Solution - Updates the Endpoint host metadata API for both the list and single endpoint to be space aware - Updates the agent status API for Endpoint to be space aware - Updates endpoint policy response API to be space aware - New FTR API integration test suite was created - included changes to some service utilities to enable loading test data per-space (cherry picked from commit 2a786b8)

kibanamachine · 2024-10-17T14:44:14Z

💚 All backports created successfully

Status	Branch	Result
✅	8.x

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

…iated supporting API to be space aware (#194312) (#196717) # Backport This will backport the following commits from `main` to `8.x`: - [[Security Solution][Endpoint] Update to Endpoint List and associated supporting API to be space aware (#194312)](#194312)  ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport)  Co-authored-by: Paul Tavares <56442535+paul-tavares@users.noreply.github.com>

paul-tavares added 8 commits September 27, 2024 10:53

Endpoint list API updated to return hosts scopped to current space

9382f7d

Enable space awareness for endpoint agent status API/Service

79db62d

Fleet: added new getByIds() service method for Agents

1b5e7e0

Change EndpointFleetServicesFactory#asInternalUser() method + adjus…

04b3089

…t endpoint context services

Host metadata API/service (single) updated to support spaces

e18c9df

Added ensureInCurrentSpace() to fleet services and refactored endpo…

0dfe39c

…int metadata to use it

Updated Policy response API to be space aware

36af78c

update to jsdoc comment

33c0bc9

paul-tavares added release_note:skip Skip the PR/issue when compiling release notes Team:Defend Workflows “EDR Workflows” sub-team of Security Solution backport:prev-minor Backport to (8.x) the previous minor version (i.e. one version back from main) v8.16.0 labels Sep 27, 2024

paul-tavares self-assigned this Sep 27, 2024

Fix calls to createInternalScopedSoClient() due to rebase from main

967b620

paul-tavares added 9 commits September 30, 2024 10:28

Fleet: tests for agent service new method

ce76ed0

Tests for agent status api handler

aab3353

Fleet: Add missing members to agent service mocks

339c569

Tests for metadata api handler

c626af6

Tests for policy response api + service

97f44fa

Merge remote-tracking branch 'upstream/main' into task/olm-8538-endpo…

e02cd6b

…int-list-api-support-spaces-2

Tests for the metadata service + mock and generator updates

373b59e

Tests for Endpoint Agent Status

6964f1a

Merge remote-tracking branch 'upstream/main' into task/olm-8538-endpo…

32ca590

…int-list-api-support-spaces-2

paul-tavares added 4 commits October 2, 2024 16:29

Fix type in policy response route registration

87f7093

Fix types in tests for policy response handlers

ec157ac

Fix API integration test

410ad02

Merge remote-tracking branch 'upstream/main' into task/olm-8538-endpo…

5a496b5

…int-list-api-support-spaces-2

paul-tavares and others added 2 commits October 8, 2024 16:04

Fix failing tests in serverless

cf4ce36

Merge branch 'main' into task/olm-8538-endpoint-list-api-support-spac…

c45b17b

…es-2

juliaElastic approved these changes Oct 9, 2024

View reviewed changes

paul-tavares and others added 2 commits October 9, 2024 10:22

Merge branch 'main' into task/olm-8538-endpoint-list-api-support-spac…

38b0a93

…es-2

Updated endpoint hosts indexer to now include the initial policy id r…

c9f8c7e

…eported by endpoint in the data indexed

paul-tavares added 2 commits October 16, 2024 08:28

Merge branch 'main' into task/olm-8538-endpoint-list-api-support-spac…

12f5b9e

…es-2

Merge branch 'main' into task/olm-8538-endpoint-list-api-support-spac…

bfbb682

…es-2

ashokaditya approved these changes Oct 16, 2024

View reviewed changes

paul-tavares and others added 6 commits October 16, 2024 15:34

Adjustments from code review

ae5c0f4

Merge remote-tracking branch 'origin/task/olm-8538-endpoint-list-api-…

4f60af2

…support-spaces-2' into task/olm-8538-endpoint-list-api-support-spaces-2

Merge remote-tracking branch 'upstream/main' into task/olm-8538-endpo…

b9b7f54

…int-list-api-support-spaces-2

Merge branch 'main' into task/olm-8538-endpoint-list-api-support-spac…

081389b

…es-2

Merge branch 'main' into task/olm-8538-endpoint-list-api-support-spac…

3b741e6

…es-2

Merge branch 'main' into task/olm-8538-endpoint-list-api-support-spac…

9ef0bea

…es-2

paul-tavares enabled auto-merge (squash) October 17, 2024 12:43

paul-tavares added v8.17.0 and removed v8.16.0 labels Oct 17, 2024

paul-tavares merged commit 2a786b8 into elastic:main Oct 17, 2024
45 checks passed

kibanamachine added the v9.0.0 label Oct 17, 2024

paul-tavares deleted the task/olm-8538-endpoint-list-api-support-spaces-2 branch October 17, 2024 14:25

kibanamachine mentioned this pull request Oct 17, 2024

[8.x] [Security Solution][Endpoint] Update to Endpoint List and associated supporting API to be space aware (#194312) #196717

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Security Solution][Endpoint] Update to Endpoint List and associated supporting API to be space aware #194312

[Security Solution][Endpoint] Update to Endpoint List and associated supporting API to be space aware #194312

paul-tavares commented Sep 27, 2024 •

edited by kibanamachine

Loading

paul-tavares commented Sep 27, 2024

paul-tavares commented Oct 2, 2024

paul-tavares commented Oct 3, 2024

paul-tavares commented Oct 9, 2024

juliaElastic left a comment

kibanamachine commented Oct 9, 2024

paul-tavares commented Oct 9, 2024

kibanamachine commented Oct 9, 2024

ashokaditya left a comment

ashokaditya Oct 16, 2024

paul-tavares Oct 16, 2024

ashokaditya Oct 16, 2024

paul-tavares Oct 16, 2024

ashokaditya Oct 16, 2024

ashokaditya Oct 16, 2024

ashokaditya Oct 16, 2024

ashokaditya Oct 16, 2024

ashokaditya Oct 16, 2024

paul-tavares Oct 16, 2024

kibanamachine commented Oct 17, 2024

elasticmachine commented Oct 17, 2024

API count

kibanamachine commented Oct 17, 2024

	router.get(
	{
	path: '/internal/spaces/_active_space',
	validate: false,
	},

[Security Solution][Endpoint] Update to Endpoint List and associated supporting API to be space aware #194312

[Security Solution][Endpoint] Update to Endpoint List and associated supporting API to be space aware #194312

Conversation

paul-tavares commented Sep 27, 2024 • edited by kibanamachine Loading

Summary

Fleet

Security Solution

Testing

Checklist

paul-tavares commented Sep 27, 2024

paul-tavares commented Oct 2, 2024

paul-tavares commented Oct 3, 2024

paul-tavares commented Oct 9, 2024

juliaElastic left a comment

Choose a reason for hiding this comment

kibanamachine commented Oct 9, 2024

Flaky Test Runner Stats

🟠 Some tests failed. - kibana-flaky-test-suite-runner#7112

paul-tavares commented Oct 9, 2024

kibanamachine commented Oct 9, 2024

Flaky Test Runner Stats

🎉 All tests passed! - kibana-flaky-test-suite-runner#7113

ashokaditya left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kibanamachine commented Oct 17, 2024

elasticmachine commented Oct 17, 2024

💚 Build Succeeded

Metrics [docs]

Public APIs missing comments

API count

History

kibanamachine commented Oct 17, 2024

💚 All backports created successfully

Questions ?

paul-tavares commented Sep 27, 2024 •

edited by kibanamachine

Loading