Enabling risk/observable matching

[cmcginley-splunk]

Context

• When risk message validation was added, an early version of risk/observable matching was added as well, but kept disabled

Code changes

• Risk/observable matching enabled
• Risks are matched against observables using the field extracted from contributing_events_search
• The following validations are performed
  • All risk events can be matched to an observable
  • All observables are matched to at least one risk event
  • All risk events have the expected typing relative to matched observable
  • All risk events can be matched to only one observable
  • All risk events match observables with valid roles only (e.g. no risk events match Attacker role observables)
    • There is one caveat w/ this validation
    • It's possible, based on timing, that a detection may pass this validation falsely
    • If at the time of checking, the offending risk event matching the Attacker observable has not yet been generated and all other risk events pass validation, then this condition will seemingly be met
    • See Refactor risk/notable querying to pin to a single savedsearch ID #248 for a possible future mitigation for this to ensure this validation is applied more reliably

TODO

• Remove comment (or annotate w/ GitHub issue) relating to having multiple conflicting roles once I get feedback
• Deploy to security_content and security-content-automation after merge
• Disable verbose logging

Testing

• Preliminary testing has shown the following true failures:
  • ~50 detections have Attacker role observables that can be matched risk events incorrectly
    • NOTE: Lou's PR Threat objects #234 has been shown to resolve these issues
  • 7 detections show risk events with improper types
    • 6 of these are due to an issue in contentctl build which causes User Name observables to create other type risk events instead of user type (see The "User Name" type should map to a "user" risk object and not "other" #246)
    • The other is also due to issues in contentctl build, which causes any observable with a role of Other (regardless of its type) to create other type risk events (see Refactor the risk property of detection_abstract to handle observable/risk/threat mappings more transparently #247)
  • 2 detections have an observable repeated by mistake
• Actionable fixes needed in ESCU are being tracked via internal tickets

Future Work

• ESCU content fixers (tracked internally)
  • repeated observable blocks
  • risk events with type other instead of user
  • observables creating risk events instead of threat objects
• The "User Name" type should map to a "user" risk object and not "other" #246
• Refactor the risk property of detection_abstract to handle observable/risk/threat mappings more transparently #247
• Refactor risk/notable querying to pin to a single savedsearch ID #248
• Add a pydantic validator to ensure observable names are unique in detection.tags.observable #249
• Re-enable and refactor code that validates the specific counts of each risk event matched to each observable #250
• Add validation and matching of threat objects to observables  #251
• Refactor and re-enable per-field validation of risk events #252

[pyth0n1c]

Initial static review with responses to RFC and followup comments. I will merge Lou's PR, update the branch, then do some runtime testing of the PR.
As such, this is not approved YET.

[pyth0n1c]

approved after discussions with dev through comments and slack