[Fleet] RBAC - Add API integration tests for bulk endpoints proceed in batches

[jillguyonnet]

With #185040 the agents write APIs were made space-aware. As part of this change, a number of API integration tests were added, mostly in https://github.com/elastic/kibana/blob/main/x-pack/test/fleet_api_integration/apis/space_awareness/agents.ts.

However, these tests don't adequately cover the case of certain bulk queries when these are processed in batches. These test cases should be added for bulk endpoints that can result in batch processing.

Example

When a kuery is passed to POST /agents/bulk_reassign and the number of agents to be processed is higher than the batch size, a task is kicked off:

kibana/x-pack/plugins/fleet/server/services/agents/reassign.ts

Lines 133 to 148 in 97b5479

	if (res.total <= batchSize) {
	givenAgents = res.agents;
	} else {
	return await new ReassignActionRunner(
	esClient,
	soClient,
	{
	...options,
	spaceId: currentNameSpace,
	batchSize,
	total: res.total,
	newAgentPolicyId,
	},
	{ pitId: await openPointInTime(esClient) }
	).runActionAsyncWithRetry();
	}

The snippet below is a WIP test case that aims to cover this scenario. Note that it only worked intermittently (usually the first run on a fresh setup passed and the following ones timed out).

Note: looking at the ES logs, it seems that the tasks are retried forever after a test fails.

it('should reassign agents in the same space by kuery in batches', async () => {
       // Initial test to check that we are starting as expected
        let agents = await apiClient.getAgents();
        let agentPolicyIds = getAgentPolicyIds(agents);
        expect(agentPolicyIds).to.eql({
          [defaultSpaceAgent1]: defaultSpacePolicy1.item.id,
          [defaultSpaceAgent2]: defaultSpacePolicy2.item.id,
        });
        agents = await apiClient.getAgents(TEST_SPACE_1);
        agentPolicyIds = getAgentPolicyIds(agents);
        expect(agentPolicyIds).to.eql({
          [testSpaceAgent1]: spaceTest1Policy1.item.id,
          [testSpaceAgent2]: spaceTest1Policy2.item.id,
          [testSpaceAgent3]: spaceTest1Policy1.item.id,
        });

       // Do a bulk reassign with a small batch size, resulting in a task running
        const res = await apiClient.bulkReassignAgents(
          {
            agents: '*',
            policy_id: spaceTest1Policy2.item.id,
            batchSize: 1,
          },
          TEST_SPACE_1
        );

       // Utility function to verify the results of reassigning
        const verifyActionResult = async () => {
          agents = await apiClient.getAgents(TEST_SPACE_1);
          agentPolicyIds = getAgentPolicyIds(agents);
          expect(agentPolicyIds).to.eql({
            [testSpaceAgent1]: spaceTest1Policy2.item.id,
            [testSpaceAgent2]: spaceTest1Policy2.item.id,
            [testSpaceAgent3]: spaceTest1Policy2.item.id,
          });
        };

       // Retry logic that waits until the agent actions is ready to verify the results
        await new Promise((resolve, reject) => {
          let attempts = 0;
          const intervalId = setInterval(async () => {
            if (attempts > 20) {
              clearInterval(intervalId);
              reject(new Error('action timed out'));
            }
            ++attempts;
            const actionStatus = await apiClient.getActionStatus(TEST_SPACE_1);
            const action = actionStatus.items.find((a: any) => a.actionId === res.actionId);
            if (action && action.nbAgentsActioned === 3 && action.nbAgentsActionCreated === 2) {
              clearInterval(intervalId);
              await verifyActionResult();
              resolve({});
            }
          }, 1000);
        }).catch((e) => {
          throw e;
        });
      });
    });

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)