-
Notifications
You must be signed in to change notification settings - Fork 8.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Endpoint] Sample data generator for endpoint app #58936
Conversation
Pinging @elastic/endpoint-app-team (Feature:Endpoint) |
Pinging @elastic/endpoint-response (Team:Endpoint Response) |
for (let i = 0; i < generations; i++) { | ||
const newParents: EndpointEvent[] = []; | ||
parents.forEach(element => { | ||
// const numChildren = randomN(maxChildrenPerNode); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
you might still be working on stuff, but probably can delete this
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
oops yeah i need to replace the random generator with a seeded random generator so we can have determinism but also a simple way to get variety
start: number; | ||
malware_classifier: MalwareClassifierFields; | ||
malware_classifier?: MalwareClassifierFields; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
just a general question, what was the criteria of making some of these fields optional?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I made as many things optional as I could without breaking the type checker
@elasticmachine merge upstream |
@elasticmachine merge upstream |
💛 Build succeeded, but was flaky
Test FailuresKibana Pipeline / kibana-xpack-agent / X-Pack Spaces API Integration Tests -- security_and_spaces.x-pack/test/spaces_api_integration/security_and_spaces/apis/resolve_copy_to_space_conflicts·ts.spaces api with security resolve copy to spaces conflicts user with no access from the default space "before each" hook for "should return 404 when overwriting, with references"Standard Out
Stack Trace
Kibana Pipeline / kibana-xpack-agent / X-Pack Spaces API Integration Tests -- security_and_spaces.x-pack/test/spaces_api_integration/security_and_spaces/apis/resolve_copy_to_space_conflicts·ts.spaces api with security resolve copy to spaces conflicts user with no access from the default space "before each" hook for "should return 404 when overwriting, with references"Standard Out
Stack Trace
Kibana Pipeline / kibana-xpack-agent / X-Pack Spaces API Integration Tests -- security_and_spaces.x-pack/test/spaces_api_integration/security_and_spaces/apis/resolve_copy_to_space_conflicts·ts.spaces api with security resolve copy to spaces conflicts user with no access from the default space "after each" hook for "should return 404 when overwriting, with references"Standard Out
Stack Trace
History
To update your PR or re-run it, just comment with: |
@@ -6,9 +6,11 @@ | |||
"license": "Elastic-License", | |||
"scripts": {}, | |||
"dependencies": { | |||
"react-redux": "^7.1.0" | |||
"react-redux": "^7.1.0", | |||
"seedrandom": "^3.0.5" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This library seems to already be included in Kibana. https://github.com/elastic/kibana/blob/master/package.json#L255
}, | ||
}, | ||
}; | ||
return generator.generateEndpointMetadata(new Date().getTime()); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You could just do generator.generateEndpointMetadata()
since you added a default date.
}, | ||
}, | ||
}; | ||
return generator.generateEndpointMetadata(new Date().getTime()); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You could just do generator.generateEndpointMetadata()
since you added a default date.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks great!
* master: Add a retry to dashboard test for a sometimes slow async operation (elastic#59600) [Endpoint] Sample data generator for endpoint app (elastic#58936) [Vis Editor] Refactoring metrics axes (elastic#59135) [DOCS] Changed Discover app to Discover (elastic#59769) [Metrics Alerts] Add support for search query and groupBy on alerts (elastic#59388) Enhancement - EUICodeEditor for Visualize JSON (elastic#58679) [ML] Transforms: Data grid fixes. (elastic#59538) [SIEM] Fix and consolidate handling of error responses in the client (elastic#59438) [Maps] convert tooltip classes to typescript (elastic#59589) [ML] Functional tests - re-activate date_nanos test (elastic#59649) Move canvas to use NP Expressions service (elastic#58387) Update misc dependencies (elastic#59542) [Unit Testing] Configure react-testing-library queries to use Kibana's data-test-subj instead of default data-testid (elastic#59445) [Console] Remove unused code (elastic#59554) [Logs / Metrics UI] Link handling / stop page reloads (elastic#58478) Add SavedObject management section registration in core (elastic#59291)
Friendly reminder: Looks like this PR hasn’t been backported yet. |
* scaffolding and notes.md * add skeleton event generator to kibana * add optional entityID param to generateEvent * add tree generation * add tests * working tests * fix up tests * fix linting * fix event types * make process parent types consistent * make generator match types * move test resolver node out of common types * fix random string generation * fix typecheck errors * remove extraneous stuff * address PR comments * add test for full resolver tree * cleanup * make tests clearer * add seedrandom to endpoint plugin. contains DONOTMERGE example code * remove robs test * start replacing random with seedrandom * use seeded random for uuidv4 * separate out IP randomization * typecheck fixes Co-authored-by: oatkiller <robert.austin@elastic.co> Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Friendly reminder: Looks like this PR hasn’t been backported yet. |
* scaffolding and notes.md * add skeleton event generator to kibana * add optional entityID param to generateEvent * add tree generation * add tests * working tests * fix up tests * fix linting * fix event types * make process parent types consistent * make generator match types * move test resolver node out of common types * fix random string generation * fix typecheck errors * remove extraneous stuff * address PR comments * add test for full resolver tree * cleanup * make tests clearer * add seedrandom to endpoint plugin. contains DONOTMERGE example code * remove robs test * start replacing random with seedrandom * use seeded random for uuidv4 * separate out IP randomization * typecheck fixes Co-authored-by: oatkiller <robert.austin@elastic.co> Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Friendly reminder: Looks like this PR hasn’t been backported yet. |
* scaffolding and notes.md * add skeleton event generator to kibana * add optional entityID param to generateEvent * add tree generation * add tests * working tests * fix up tests * fix linting * fix event types * make process parent types consistent * make generator match types * move test resolver node out of common types * fix random string generation * fix typecheck errors * remove extraneous stuff * address PR comments * add test for full resolver tree * cleanup * make tests clearer * add seedrandom to endpoint plugin. contains DONOTMERGE example code * remove robs test * start replacing random with seedrandom * use seeded random for uuidv4 * separate out IP randomization * typecheck fixes Co-authored-by: oatkiller <robert.austin@elastic.co> Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Looks like this PR has a backport PR but it still hasn't been merged. Please merge it ASAP to keep the branches relatively in sync. |
1 similar comment
Looks like this PR has a backport PR but it still hasn't been merged. Please merge it ASAP to keep the branches relatively in sync. |
* scaffolding and notes.md * add skeleton event generator to kibana * add optional entityID param to generateEvent * add tree generation * add tests * working tests * fix up tests * fix linting * fix event types * make process parent types consistent * make generator match types * move test resolver node out of common types * fix random string generation * fix typecheck errors * remove extraneous stuff * address PR comments * add test for full resolver tree * cleanup * make tests clearer * add seedrandom to endpoint plugin. contains DONOTMERGE example code * remove robs test * start replacing random with seedrandom * use seeded random for uuidv4 * separate out IP randomization * typecheck fixes Co-authored-by: oatkiller <robert.austin@elastic.co> Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com> Co-authored-by: oatkiller <robert.austin@elastic.co> Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Summary
This PR adds a generator that provides sample events, alerts, and endpoint metadata documents for use in testing. Documents created using an instance of the generator will have corresponding host and agent information so they can be correlated. Additionally, helper functions
generateResolverTree
andgenerateEventAncestry
automatically create a set of events that have parent-child relationships defined by their entity_ids.generateResolverTree
expects a root event to be passed in and generates and returns a list of events and alerts that form the tree of children as well as related events for some random child processes.Checklist
Delete any items that are not applicable to this PR.
For maintainers