-
Notifications
You must be signed in to change notification settings - Fork 8.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Endpoint] Sample data generator for endpoint app #58936
Merged
Merged
Changes from 20 commits
Commits
Show all changes
31 commits
Select commit
Hold shift + click to select a range
be66b5b
scaffolding and notes.md
fc0279f
add skeleton event generator to kibana
marshallmain 1655df5
add optional entityID param to generateEvent
marshallmain 9c0b912
add tree generation
marshallmain 52efc27
add tests
marshallmain 5073fdc
working tests
marshallmain c1bf5d8
fix up tests
marshallmain 69a9221
merge upstream
marshallmain bdfd9b4
fix linting
marshallmain 027929a
fix event types
marshallmain c333e87
make process parent types consistent
marshallmain cf96f10
make generator match types
marshallmain 4bc37d4
move test resolver node out of common types
marshallmain bf6a055
fix random string generation
marshallmain f82ef50
merge upstream
marshallmain 8a6bcbf
fix typecheck errors
marshallmain bfd320f
remove extraneous stuff
marshallmain 9bf2bf3
address PR comments
marshallmain e1f6df2
add test for full resolver tree
marshallmain de3b9b7
cleanup
marshallmain 3ead579
make tests clearer
marshallmain 07fa69c
Merge branch 'master' into data-generator
marshallmain 7006c4f
add seedrandom to endpoint plugin. contains DONOTMERGE example code
d1df9fd
remove robs test
marshallmain 1ee15a0
start replacing random with seedrandom
marshallmain 5f38e3a
Merge branch 'data-generator' of github.com:marshallmain/kibana into …
marshallmain aa77f8d
use seeded random for uuidv4
marshallmain ba621f6
separate out IP randomization
marshallmain 041a34f
typecheck fixes
marshallmain 469418e
Merge branch 'master' into data-generator
elasticmachine 3f16642
Merge branch 'master' into data-generator
elasticmachine File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,140 @@ | ||
/* | ||
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one | ||
* or more contributor license agreements. Licensed under the Elastic License; | ||
* you may not use this file except in compliance with the Elastic License. | ||
*/ | ||
import { EndpointDocGenerator, Event } from './generate_data'; | ||
|
||
interface Node { | ||
events: Event[]; | ||
children: Node[]; | ||
parent_entity_id?: string; | ||
} | ||
|
||
describe('data generator', () => { | ||
let generator: EndpointDocGenerator; | ||
beforeEach(() => { | ||
generator = new EndpointDocGenerator(); | ||
}); | ||
|
||
it('creates endpoint metadata documents', () => { | ||
const timestamp = new Date().getTime(); | ||
const metadata = generator.generateEndpointMetadata(timestamp); | ||
expect(metadata['@timestamp']).toEqual(timestamp); | ||
expect(metadata.event.created).toEqual(timestamp); | ||
expect(metadata.endpoint).not.toBeNull(); | ||
expect(metadata.agent).not.toBeNull(); | ||
expect(metadata.host).not.toBeNull(); | ||
}); | ||
|
||
it('creates alert event documents', () => { | ||
const timestamp = new Date().getTime(); | ||
const alert = generator.generateAlert(timestamp); | ||
expect(alert['@timestamp']).toEqual(timestamp); | ||
expect(alert.event.action).not.toBeNull(); | ||
expect(alert.endpoint).not.toBeNull(); | ||
expect(alert.agent).not.toBeNull(); | ||
expect(alert.host).not.toBeNull(); | ||
expect(alert.process.entity_id).not.toBeNull(); | ||
}); | ||
|
||
it('creates process event documents', () => { | ||
const timestamp = new Date().getTime(); | ||
const processEvent = generator.generateEvent({ timestamp }); | ||
expect(processEvent['@timestamp']).toEqual(timestamp); | ||
expect(processEvent.event.category).toEqual('process'); | ||
expect(processEvent.event.kind).toEqual('event'); | ||
expect(processEvent.event.type).toEqual('creation'); | ||
expect(processEvent.agent).not.toBeNull(); | ||
expect(processEvent.host).not.toBeNull(); | ||
expect(processEvent.process.entity_id).not.toBeNull(); | ||
}); | ||
|
||
it('creates other event documents', () => { | ||
const timestamp = new Date().getTime(); | ||
const processEvent = generator.generateEvent({ timestamp, eventCategory: 'dns' }); | ||
expect(processEvent['@timestamp']).toEqual(timestamp); | ||
expect(processEvent.event.category).toEqual('dns'); | ||
expect(processEvent.event.kind).toEqual('event'); | ||
expect(processEvent.event.type).toEqual('creation'); | ||
expect(processEvent.agent).not.toBeNull(); | ||
expect(processEvent.host).not.toBeNull(); | ||
expect(processEvent.process.entity_id).not.toBeNull(); | ||
}); | ||
|
||
it('creates alert ancestor tree', () => { | ||
const events = generator.generateAlertEventAncestry(3); | ||
for (let i = 1; i < events.length - 1; i++) { | ||
expect(events[i].process.parent?.entity_id).toEqual(events[i - 1].process.entity_id); | ||
expect(events[i].event.kind).toEqual('event'); | ||
expect(events[i].event.category).toEqual('process'); | ||
} | ||
// The alert should be last and have the same entity_id as the previous process event | ||
expect(events[events.length - 1].process.entity_id).toEqual( | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. it would probably be better to break some of these assertions into separate
|
||
events[events.length - 2].process.entity_id | ||
); | ||
expect(events[events.length - 1].process.parent?.entity_id).toEqual( | ||
events[events.length - 2].process.parent?.entity_id | ||
); | ||
expect(events[events.length - 1].event.kind).toEqual('alert'); | ||
expect(events[events.length - 1].event.category).toEqual('malware'); | ||
}); | ||
|
||
function buildResolverTree(events: Event[]): Node { | ||
// First pass we gather up all the events by entity_id | ||
const tree: Record<string, Node> = {}; | ||
events.forEach(event => { | ||
if (event.process.entity_id in tree) { | ||
tree[event.process.entity_id].events.push(event); | ||
} else { | ||
tree[event.process.entity_id] = { | ||
events: [event], | ||
children: [], | ||
parent_entity_id: event.process.parent?.entity_id, | ||
}; | ||
} | ||
}); | ||
// Second pass add child references to each node | ||
for (const value of Object.values(tree)) { | ||
if (value.parent_entity_id) { | ||
tree[value.parent_entity_id].children.push(value); | ||
} | ||
} | ||
// The root node must be first in the array or this fails | ||
return tree[events[0].process.entity_id]; | ||
} | ||
|
||
function countResolverEvents(rootNode: Node, generations: number): number { | ||
// Start at the root, traverse N levels of the tree and check that we found all nodes | ||
let nodes = [rootNode]; | ||
let visitedEvents = 0; | ||
for (let i = 0; i < generations + 1; i++) { | ||
let nextNodes: Node[] = []; | ||
nodes.forEach(node => { | ||
nextNodes = nextNodes.concat(node.children); | ||
visitedEvents += node.events.length; | ||
}); | ||
nodes = nextNodes; | ||
} | ||
return visitedEvents; | ||
} | ||
|
||
it('creates tree of process children', () => { | ||
const timestamp = new Date().getTime(); | ||
const root = generator.generateEvent({ timestamp }); | ||
const generations = 2; | ||
const events = generator.generateDescendantsTree(root, generations); | ||
const rootNode = buildResolverTree(events); | ||
const visitedEvents = countResolverEvents(rootNode, generations); | ||
expect(visitedEvents).toEqual(events.length); | ||
}); | ||
|
||
it('creates full resolver tree', () => { | ||
const alertAncestors = 3; | ||
const generations = 2; | ||
const events = generator.generateFullResolverTree(alertAncestors, generations); | ||
const rootNode = buildResolverTree(events); | ||
const visitedEvents = countResolverEvents(rootNode, alertAncestors + generations); | ||
expect(visitedEvents).toEqual(events.length); | ||
}); | ||
}); |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
in reference to the later comment about 'snapshot' testing, you could try this instead:
Save the file, then run the tests (or if you're using
--watch
, no need.) The test framework will serializeprocessEvent
and drop a serialized version into this file. If the serialization changes, the test will fail. Not the best for every type of test, but in this case it might be worth trying out.Another thing, if the serialization is really long/big, you can write
toMatchSnapshot()
instead and the serialization will be saved in a separate file.One possible downside to using snapshots is that the intent of the assertion can be lost to the reader. In your test here for example, I can tell that the
'@timestamp'
field should match thetimestamp
variable, but that might not be obvious if there is just a single snapshot assertion. In this case, you can make the test description longer, like 'it creates process events, using the passed-in timestamp to populate timestamp fields.'