[Security Solutions] Exposes the search_after and point in time (pit)…

… from saved objects to exception lists (#125182)

## Summary

Exposes the functionality of
* search_after
* point in time (pit)

From saved objects to the exception lists. This _DOES NOT_ expose these to the REST API just yet. Rather this exposes it at the API level to start with and changes code that had hard limits of 10k and other limited loops. I use the batching of 1k for this at a time as I thought that would be a decent batch guess and I see other parts of the code changed to it. It's easy to change the 1k if we find we need to throttle back more as we get feedback from others.

See this PR where `PIT` and `search_after` were first introduced: #89915
See these 2 issues where we should be using more paging and PIT (Point in Time) with search_after: #93770 #103944

The new methods added to the `exception_list_client.ts` client class are:
* openPointInTime
* closePointInTime
* findExceptionListItemPointInTimeFinder
* findExceptionListPointInTimeFinder
* findExceptionListsItemPointInTimeFinder
* findValueListExceptionListItemsPointInTimeFinder

The areas of functionality that have been changed:
* Exception list exports
* Deletion of lists
* Getting exception list items when generating signals

Note that currently we use our own ways of looping over the saved objects which you can see in the codebase such as this older way below which does work but had a limitation of 10k against saved objects and did not do point in time (PIT)

Older way example (deprecated):
```ts
  let page = 1;
  let ids: string[] = [];
  let foundExceptionListItems = await findExceptionListItem({
    filter: undefined,
    listId,
    namespaceType,
    page,
    perPage: PER_PAGE,
    pit: undefined,
    savedObjectsClient,
    searchAfter: undefined,
    sortField: 'tie_breaker_id',
    sortOrder: 'desc',
  });
  while (foundExceptionListItems != null && foundExceptionListItems.data.length > 0) {
    ids = [
      ...ids,
      ...foundExceptionListItems.data.map((exceptionListItem) => exceptionListItem.id),
    ];
    page += 1;
    foundExceptionListItems = await findExceptionListItem({
      filter: undefined,
      listId,
      namespaceType,
      page,
      perPage: PER_PAGE,
      pit: undefined,
      savedObjectsClient,
      searchAfter: undefined,
      sortField: 'tie_breaker_id',
      sortOrder: 'desc',
    });
  }
  return ids;
```

But now that is replaced with this newer way using PIT:
```ts
  // Stream the results from the Point In Time (PIT) finder into this array
  let ids: string[] = [];
  const executeFunctionOnStream = (response: FoundExceptionListItemSchema): void => {
    const responseIds = response.data.map((exceptionListItem) => exceptionListItem.id);
    ids = [...ids, ...responseIds];
  };

  await findExceptionListItemPointInTimeFinder({
    executeFunctionOnStream,
    filter: undefined,
    listId,
    maxSize: undefined, // NOTE: This is unbounded when it is "undefined"
    namespaceType,
    perPage: 1_000,
    savedObjectsClient,
    sortField: 'tie_breaker_id',
    sortOrder: 'desc',
  });
  return ids;
```

We also have areas of code that has perPage listed at 10k or a constant that represents 10k which this removes in most areas (but not all areas):
```ts
      const items = await client.findExceptionListsItem({
        listId: listIds,
        namespaceType: namespaceTypes,
        page: 1,
        pit: undefined,
        perPage: MAX_EXCEPTION_LIST_SIZE, // <--- Really bad to send in 10k per page at a time
        searchAfter: undefined,
        filter: [],
        sortOrder: undefined,
        sortField: undefined,
      });
```

That is now:
```ts
      // Stream the results from the Point In Time (PIT) finder into this array
      let items: ExceptionListItemSchema[] = [];
      const executeFunctionOnStream = (response: FoundExceptionListItemSchema): void => {
        items = [...items, ...response.data];
      };

      await client.findExceptionListsItemPointInTimeFinder({
        executeFunctionOnStream,
        listId: listIds,
        namespaceType: namespaceTypes,
        perPage: 1_000,
        filter: [],
        maxSize: undefined, // NOTE: This is unbounded when it is "undefined"
        sortOrder: undefined,
        sortField: undefined,
      });
```

Left over areas will be handled in separate PR's because they are in other people's code ownership areas.

### Checklist
- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios

packages/kbn-securitysolution-io-ts-list-types/src/common/index.ts

@@ -35,13 +35,16 @@ export * from './list_operator';
 export * from './list_type';
 export * from './lists';
 export * from './lists_default_array';
+export * from './max_size';
 export * from './meta';
 export * from './name';
 export * from './non_empty_entries_array';
 export * from './non_empty_nested_entries_array';
 export * from './os_type';
 export * from './page';
 export * from './per_page';
+export * from './pit';
+export * from './search_after';
 export * from './serializer';
 export * from './sort_field';
 export * from './sort_order';

packages/kbn-securitysolution-io-ts-list-types/src/common/max_size/index.test.ts

@@ -0,0 +1,59 @@
+/*
+ * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+ * or more contributor license agreements. Licensed under the Elastic License
+ * 2.0 and the Server Side Public License, v 1; you may not use this file except
+ * in compliance with, at your election, the Elastic License 2.0 or the Server
+ * Side Public License, v 1.
+ */
+
+import { exactCheck } from '@kbn/securitysolution-io-ts-utils';
+import { maxSizeOrUndefined } from '.';
+
+import { pipe } from 'fp-ts/lib/pipeable';
+import { left } from 'fp-ts/lib/Either';
+
+import { foldLeftRight, getPaths } from '@kbn/securitysolution-io-ts-utils';
+
+describe('maxSizeOrUndefined', () => {
+  test('it will validate a correct max value', () => {
+    const payload = 123;
+    const decoded = maxSizeOrUndefined.decode(payload);
+    const checked = exactCheck(payload, decoded);
+    const message = pipe(checked, foldLeftRight);
+    expect(getPaths(left(message.errors))).toEqual([]);
+    expect(message.schema).toEqual(payload);
+  });
+
+  test('it will fail to validate a 0', () => {
+    const payload = 0;
+    const decoded = maxSizeOrUndefined.decode(payload);
+    const checked = exactCheck(payload, decoded);
+    const message = pipe(checked, foldLeftRight);
+    expect(getPaths(left(message.errors))).toEqual([
+      'Invalid value "0" supplied to "(PositiveIntegerGreaterThanZero | undefined)"',
+    ]);
+    expect(message.schema).toEqual({});
+  });
+
+  test('it will fail to validate a -1', () => {
+    const payload = -1;
+    const decoded = maxSizeOrUndefined.decode(payload);
+    const checked = exactCheck(payload, decoded);
+    const message = pipe(checked, foldLeftRight);
+    expect(getPaths(left(message.errors))).toEqual([
+      'Invalid value "-1" supplied to "(PositiveIntegerGreaterThanZero | undefined)"',
+    ]);
+    expect(message.schema).toEqual({});
+  });
+
+  test('it will fail to validate a string', () => {
+    const payload = '123';
+    const decoded = maxSizeOrUndefined.decode(payload);
+    const checked = exactCheck(payload, decoded);
+    const message = pipe(checked, foldLeftRight);
+    expect(getPaths(left(message.errors))).toEqual([
+      'Invalid value "123" supplied to "(PositiveIntegerGreaterThanZero | undefined)"',
+    ]);
+    expect(message.schema).toEqual({});
+  });
+});

packages/kbn-securitysolution-io-ts-list-types/src/common/max_size/index.ts

@@ -0,0 +1,18 @@
+/*
+ * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+ * or more contributor license agreements. Licensed under the Elastic License
+ * 2.0 and the Server Side Public License, v 1; you may not use this file except
+ * in compliance with, at your election, the Elastic License 2.0 or the Server
+ * Side Public License, v 1.
+ */
+
+/* eslint-disable @typescript-eslint/naming-convention */
+
+import { PositiveIntegerGreaterThanZero } from '@kbn/securitysolution-io-ts-types';
+import * as t from 'io-ts';
+
+export const max_size = PositiveIntegerGreaterThanZero;
+export type MaxSize = t.TypeOf<typeof max_size>;
+
+export const maxSizeOrUndefined = t.union([max_size, t.undefined]);
+export type MaxSizeOrUndefined = t.TypeOf<typeof maxSizeOrUndefined>;

packages/kbn-securitysolution-io-ts-list-types/src/common/pit/index.test.ts

@@ -0,0 +1,65 @@
+/*
+ * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+ * or more contributor license agreements. Licensed under the Elastic License
+ * 2.0 and the Server Side Public License, v 1; you may not use this file except
+ * in compliance with, at your election, the Elastic License 2.0 or the Server
+ * Side Public License, v 1.
+ */
+
+import { exactCheck } from '@kbn/securitysolution-io-ts-utils';
+import { pitOrUndefined } from '.';
+
+import * as t from 'io-ts';
+import { pipe } from 'fp-ts/lib/pipeable';
+import { left } from 'fp-ts/lib/Either';
+
+import { foldLeftRight, getPaths } from '@kbn/securitysolution-io-ts-utils';
+
+describe('pitOrUndefined', () => {
+  test('it will validate a correct pit', () => {
+    const payload = { id: '123', keepAlive: '1m' };
+    const decoded = pitOrUndefined.decode(payload);
+    const checked = exactCheck(payload, decoded);
+    const message = pipe(checked, foldLeftRight);
+    expect(getPaths(left(message.errors))).toEqual([]);
+    expect(message.schema).toEqual(payload);
+  });
+
+  test('it will validate with the value of "undefined"', () => {
+    const obj = t.exact(
+      t.type({
+        pit_id: pitOrUndefined,
+      })
+    );
+    const payload: t.TypeOf<typeof obj> = {
+      pit_id: undefined,
+    };
+    const decoded = obj.decode({
+      pit_id: undefined,
+    });
+    const checked = exactCheck(payload, decoded);
+    const message = pipe(checked, foldLeftRight);
+    expect(getPaths(left(message.errors))).toEqual([]);
+    expect(message.schema).toEqual(payload);
+  });
+
+  test('it will validate a correct pit without having a "keepAlive"', () => {
+    const payload = { id: '123' };
+    const decoded = pitOrUndefined.decode(payload);
+    const checked = exactCheck(payload, decoded);
+    const message = pipe(checked, foldLeftRight);
+    expect(getPaths(left(message.errors))).toEqual([]);
+    expect(message.schema).toEqual(payload);
+  });
+
+  test('it will fail to validate an incorrect pit', () => {
+    const payload = 'foo';
+    const decoded = pitOrUndefined.decode(payload);
+    const checked = exactCheck(payload, decoded);
+    const message = pipe(checked, foldLeftRight);
+    expect(getPaths(left(message.errors))).toEqual([
+      'Invalid value "foo" supplied to "({| id: string, keepAlive: (string | undefined) |} | undefined)"',
+    ]);
+    expect(message.schema).toEqual({});
+  });
+});

packages/kbn-securitysolution-io-ts-list-types/src/common/pit/index.ts

@@ -0,0 +1,22 @@
+/*
+ * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+ * or more contributor license agreements. Licensed under the Elastic License
+ * 2.0 and the Server Side Public License, v 1; you may not use this file except
+ * in compliance with, at your election, the Elastic License 2.0 or the Server
+ * Side Public License, v 1.
+ */
+
+import * as t from 'io-ts';
+
+export const pitId = t.string;
+export const pit = t.exact(
+  t.type({
+    id: pitId,
+    keepAlive: t.union([t.string, t.undefined]),
+  })
+);
+export const pitOrUndefined = t.union([pit, t.undefined]);
+
+export type Pit = t.TypeOf<typeof pit>;
+export type PitId = t.TypeOf<typeof pitId>;
+export type PitOrUndefined = t.TypeOf<typeof pitOrUndefined>;

packages/kbn-securitysolution-io-ts-list-types/src/common/search_after/index.test.ts

@@ -0,0 +1,56 @@
+/*
+ * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+ * or more contributor license agreements. Licensed under the Elastic License
+ * 2.0 and the Server Side Public License, v 1; you may not use this file except
+ * in compliance with, at your election, the Elastic License 2.0 or the Server
+ * Side Public License, v 1.
+ */
+
+import { exactCheck } from '@kbn/securitysolution-io-ts-utils';
+import { searchAfterOrUndefined } from '.';
+
+import * as t from 'io-ts';
+import { pipe } from 'fp-ts/lib/pipeable';
+import { left } from 'fp-ts/lib/Either';
+
+import { foldLeftRight, getPaths } from '@kbn/securitysolution-io-ts-utils';
+
+describe('searchAfter', () => {
+  test('it will validate a correct search_after', () => {
+    const payload = ['test-1', 'test-2'];
+    const decoded = searchAfterOrUndefined.decode(payload);
+    const checked = exactCheck(payload, decoded);
+    const message = pipe(checked, foldLeftRight);
+    expect(getPaths(left(message.errors))).toEqual([]);
+    expect(message.schema).toEqual(payload);
+  });
+
+  test('it will validate with the value of "undefined"', () => {
+    const obj = t.exact(
+      t.type({
+        search_after: searchAfterOrUndefined,
+      })
+    );
+    const payload: t.TypeOf<typeof obj> = {
+      search_after: undefined,
+    };
+    const decoded = obj.decode({
+      pit_id: undefined,
+    });
+    const checked = exactCheck(payload, decoded);
+    const message = pipe(checked, foldLeftRight);
+    expect(getPaths(left(message.errors))).toEqual([]);
+    expect(message.schema).toEqual(payload);
+  });
+
+  test('it will fail to validate an incorrect search_after', () => {
+    const payload = 'foo';
+    const decoded = searchAfterOrUndefined.decode(payload);
+    const checked = exactCheck(payload, decoded);
+    const message = pipe(checked, foldLeftRight);
+    expect(getPaths(left(message.errors))).toEqual([
+      'Invalid value "foo" supplied to "(Array<string> | undefined)"',
+    ]);
+    expect(message.schema).toEqual({});
+  });
+});

packages/kbn-securitysolution-io-ts-list-types/src/common/search_after/index.ts

@@ -0,0 +1,17 @@
+/*
+ * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+ * or more contributor license agreements. Licensed under the Elastic License
+ * 2.0 and the Server Side Public License, v 1; you may not use this file except
+ * in compliance with, at your election, the Elastic License 2.0 or the Server
+ * Side Public License, v 1.
+ */
+
+/* eslint-disable @typescript-eslint/naming-convention */
+
+import * as t from 'io-ts';
+
+export const search_after = t.array(t.string);
+export type SearchAfter = t.TypeOf<typeof search_after>;
+
+export const searchAfterOrUndefined = t.union([search_after, t.undefined]);
+export type SearchAfterOrUndefined = t.TypeOf<typeof searchAfterOrUndefined>;

packages/kbn-securitysolution-io-ts-list-types/src/response/found_exception_list_item_schema/index.ts

@@ -10,16 +10,24 @@ import * as t from 'io-ts';
 
 import { page } from '../../common/page';
 import { per_page } from '../../common/per_page';
+import { pitId } from '../../common/pit';
 import { total } from '../../common/total';
 import { exceptionListItemSchema } from '../exception_list_item_schema';
 
-export const foundExceptionListItemSchema = t.exact(
-  t.type({
-    data: t.array(exceptionListItemSchema),
-    page,
-    per_page,
-    total,
-  })
-);
+export const foundExceptionListItemSchema = t.intersection([
+  t.exact(
+    t.type({
+      data: t.array(exceptionListItemSchema),
+      page,
+      per_page,
+      total,
+    })
+  ),
+  t.exact(
+    t.partial({
+      pit: pitId,
+    })
+  ),
+]);
 
 export type FoundExceptionListItemSchema = t.TypeOf<typeof foundExceptionListItemSchema>;

packages/kbn-securitysolution-io-ts-list-types/src/response/found_exception_list_schema/index.ts

@@ -10,17 +10,21 @@ import * as t from 'io-ts';
 
 import { page } from '../../common/page';
 import { per_page } from '../../common/per_page';
+import { pitId } from '../../common/pit';
 import { total } from '../../common/total';
 
 import { exceptionListSchema } from '../exception_list_schema';
 
-export const foundExceptionListSchema = t.exact(
-  t.type({
-    data: t.array(exceptionListSchema),
-    page,
-    per_page,
-    total,
-  })
-);
+export const foundExceptionListSchema = t.intersection([
+  t.exact(
+    t.type({
+      data: t.array(exceptionListSchema),
+      page,
+      per_page,
+      total,
+    })
+  ),
+  t.exact(t.partial({ pit: pitId })),
+]);
 
 export type FoundExceptionListSchema = t.TypeOf<typeof foundExceptionListSchema>;