-
-
Notifications
You must be signed in to change notification settings - Fork 16
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Overall, this looks pretty nice. More details below.
STIX shifter error message upon CTRL+C
When hitting CTRL+C in the bridge, I get this error:
Unhandled error in exception handler
context: {'message': 'Task exception was never retrieved', 'exception': KeyboardInterrupt(), 'future': <Task finished name='Task-1' coro=<start() done, defined at /Users/mavam/code/tenzir/threatbus/apps/stix-shifter/stix_shifter_threatbus/shifter.py:186> exception=KeyboardInterrupt()>}
Traceback (most recent call last):
File "/usr/local/Cellar/python@3.8/3.8.9/Frameworks/Python.framework/Versions/3.8/lib/python3.8/asyncio/base_events.py", line 1744, in call_exception_handler
self._exception_handler(self, context)
File "/Users/mavam/code/tenzir/threatbus/apps/stix-shifter/stix_shifter_threatbus/shifter.py", line 221, in exception_handler
logger.error(f"Error in async task: {context}")
File "/usr/local/Cellar/python@3.8/3.8.9/Frameworks/Python.framework/Versions/3.8/lib/python3.8/logging/__init__.py", line 1475, in error
self._log(ERROR, msg, args, **kwargs)
File "/usr/local/Cellar/python@3.8/3.8.9/Frameworks/Python.framework/Versions/3.8/lib/python3.8/logging/__init__.py", line 1589, in _log
self.handle(record)
File "/usr/local/Cellar/python@3.8/3.8.9/Frameworks/Python.framework/Versions/3.8/lib/python3.8/logging/__init__.py", line 1599, in handle
self.callHandlers(record)
File "/usr/local/Cellar/python@3.8/3.8.9/Frameworks/Python.framework/Versions/3.8/lib/python3.8/logging/__init__.py", line 1661, in callHandlers
hdlr.handle(record)
File "/usr/local/Cellar/python@3.8/3.8.9/Frameworks/Python.framework/Versions/3.8/lib/python3.8/logging/__init__.py", line 954, in handle
self.emit(record)
File "/usr/local/Cellar/python@3.8/3.8.9/Frameworks/Python.framework/Versions/3.8/lib/python3.8/logging/__init__.py", line 1186, in emit
self.stream = self._open()
File "/usr/local/Cellar/python@3.8/3.8.9/Frameworks/Python.framework/Versions/3.8/lib/python3.8/logging/__init__.py", line 1176, in _open
return open(self.baseFilename, self.mode, encoding=self.encoding)
NameError: name 'open' is not defined
Is this something we can try-catch?
splunk query
I'm getting this output in Threat Bus:
2021-05-18 09:50:26 DEBUG [stix-shifter-threatbus] Calling Threat Bus management endpoint localhost:13370
2021-05-18 09:50:26 INFO [stix-shifter-threatbus] Subscribing to topic 'stix2/indicator'...
2021-05-18 09:50:26 INFO [stix-shifter-threatbus] Subscription successful. New p2p_topic: nveqctylybxlqxtijrorbrmwnkwmievl
2021-05-18 09:50:26 INFO [stix-shifter-threatbus] Receiving via ZMQ on topic 127.0.0.1:13371/nveqctylybxlqxtijrorbrmwnkwmievl
2021-05-18 09:50:27 DEBUG [stix-shifter-threatbus] Converting indicator from Threat Bus to module-specific query: {
"type": "indicator",
"spec_version": "2.1",
"id": "indicator--783f7d87-afd0-4653-8b97-0f7567c127c6",
"created": "2021-05-18T09:48:57.000Z",
"modified": "2021-05-18T07:50:26.936642Z",
"pattern": "[ipv4-addr:value = '198.189.255.75']",
"pattern_type": "stix",
"pattern_version": "2.1",
"valid_from": "2021-05-18T07:50:26.936642Z"
}
2021-05-18 09:50:27 DEBUG [stix-shifter-threatbus] Translated pattern to splunk query: {'queries': ['search ((src_ip = "198.189.255.75") OR (dest_ip = "198.189.255.75")) earliest="-5minutes" | head 10000 | fields src_ip, src_port, src_mac, src_ipv6, dest_ip, dest_port, dest_mac, dest_ipv6, file_hash, user, url, protocol']}
But not more. The first search
works and delivers results. But the entire query doesn't. The problem is earliest="-5minutes"
. Once I manually add this to my splunk search, I get no more results.
App Template
I would be nice to make the fields that should be customized stand out more clearly, e.g., version
etc.
Co-authored-by: Matthias Vallentin <matthias@tenzir.com>
@mavam I added logic for gracefully shutting down all ZeroMQ apps, that should elimitnate your stacktrace when hitting
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Works as expected now. Really nice work!
📔 Description
This PR adds a new app that connects with Threat Bus via ZeroMQ and subscribes to the topic
stix2/indicator
. When receiving indicators, the app calls out to the STIX-Shifter library to transform the STIX-2 Indicator to a specific query, depending on the user-provided configuration file. Users can configure themodules
section in theconfig.yaml
file. That section instructs the app to call out to certain STIX-Shifter modules, e.g., Splunk or Qradar.This PR also extracts a template to create Python applications that connect to Threat Bus via ZeroMQ.
📝 Checklist
🎯 Review Instructions
Commit-by-commit
I tested this interactively with Splunk. The example config works for local Splunk installations. Grab the trial version from the official pages and start it. Then start Threat Bus with the zmq-app plugin enabled as usual and finally start this app. To see results, you first need to ingest data into Splunk (e.g., by uploading a file). Once that is done, you can use the
tests/utils/zmq_sender.py
to send specially crafted indicators (patterns) to match you data in Splunk. A pattern that worked well for me ispattern="[ipv4-addr:value = '192.168.1.103'] START t'2008-10-08T23:59:00.000Z' STOP t'2021-10-08T23:59:00.000Z'"
.Unit / integration tests will be part of another PR. Same goes for the actual processing of result data.