Forward indicators to STIX-Shifter

CHANGELOG.md

@@ -23,6 +23,11 @@ Every entry has a category for which we use the following visual abbreviations:
   Threat Bus. Starting without it will print a helpful error message.
   [#119](https://github.com/tenzir/threatbus/pull/119)
 
+- 🎁 We now provide a simple asyncio
+  [template](https://github.com/tenzir/threatbus/tree/master/apps/zmq-app-template)
+  for writing applications that connect to Threat Bus via ZeroMQ.
+  [#118](https://github.com/tenzir/threatbus/pull/118)
+
 - ⚠️ The `threatbus-zeek` plugin now uses the timestamp of Zeek intel matches to
   set the `last_seen` property of resulting STIX-2 Sightings, instead of setting
   the `created` timestamp. The `created` timestamp now always refers to the

Makefile

@@ -25,6 +25,7 @@ unit-tests:
 	$(MAKE) -C plugins/apps/threatbus_zeek unit-tests
 	$(MAKE) -C plugins/apps/threatbus_cif3 unit-tests
 	$(MAKE) -C apps/vast unit-tests
+	$(MAKE) -C apps/stix-shifter unit-tests
 
 .PHONY: integration-tests
 integration-tests:
@@ -50,6 +51,7 @@ clean:
 	-$(MAKE) -C plugins/backbones/threatbus_inmem clean
 	-$(MAKE) -C plugins/backbones/threatbus_rabbitmq clean
 	-$(MAKE) -C apps/vast clean
+	-$(MAKE) -C apps/stix-shifter clean
 
 .PHONY: build
 build:
@@ -61,6 +63,7 @@ build:
 	$(MAKE) -C plugins/backbones/threatbus_inmem build
 	$(MAKE) -C plugins/backbones/threatbus_rabbitmq build
 	$(MAKE) -C apps/vast build
+	$(MAKE) -C apps/stix-shifter build
 
 .PHONY: dist
 dist:
@@ -72,6 +75,7 @@ dist:
 	$(MAKE) -C plugins/backbones/threatbus_inmem dist
 	$(MAKE) -C plugins/backbones/threatbus_rabbitmq dist
 	$(MAKE) -C apps/vast dist
+	$(MAKE) -C apps/stix-shifter dist
 
 .PHONY: install
 install:
@@ -83,6 +87,7 @@ install:
 	$(MAKE) -C plugins/backbones/threatbus_inmem install
 	$(MAKE) -C plugins/backbones/threatbus_rabbitmq install
 	$(MAKE) -C apps/vast install
+	$(MAKE) -C apps/stix-shifter install
 
 .PHONY: dev-mode
 dev-mode:
@@ -92,5 +97,6 @@ dev-mode:
 	$(MAKE) -C plugins/backbones/threatbus_rabbitmq dev-mode
 	$(MAKE) -C plugins/apps/threatbus_misp dev-mode
 	$(MAKE) -C plugins/apps/threatbus_zeek dev-mode
-	$(MAKE) -C apps/vast dev-mode
 	$(MAKE) -C plugins/apps/threatbus_cif3 dev-mode
+	$(MAKE) -C apps/vast dev-mode
+	$(MAKE) -C apps/stix-shifter dev-mode

apps/stix-shifter/CHANGELOG.md

@@ -0,0 +1,24 @@
+# Changelog
+
+This changelog documents all notable user-facing changes of
+`stix-shifter-threatbus`.
+
+Every entry has a category for which we use the following visual abbreviations:
+
+- 🎁 Features
+- 🧬 Experimental Features
+- ⚠️ Changes
+- ⚡️ Breaking Changes
+- 🐞 Bug Fixes
+
+## Unreleased
+
+- 🎁 `stix-shifter-threatbus` has come to life. This stand-alone application
+  connects to Threat Bus via ZeroMQ and bridges the gap between Threat Bus and
+  commercial security tools, like
+  [IBM QRadar](https://www.ibm.com/security/security-intelligence/qradar) or
+  [Splunk](https://www.splunk.com/). `stix-shifter-threatbus` uses
+  [STIX-Shifter](https://github.com/opencybersecurityalliance/stix-shifter) to
+  first translate STIX-2 Indicators to native queries for commercial tools and
+  then execute these queries to log the results.
+  [#118](https://github.com/tenzir/threatbus/pull/118)

apps/stix-shifter/Makefile

@@ -0,0 +1,36 @@
+colon := :
+$(colon) := :
+
+.PHONY: all
+all: format build dist test
+
+.PHONY: format
+format:
+	python -m black .
+
+.PHONY: test
+test: unit-tests
+
+.PHONY: unit-tests
+unit-tests:
+	python -m unittest discover .
+
+.PHONY: clean
+clean:
+	${RM} -r __pycache__ *egg-info build dist
+
+.PHONY: build
+build:
+	python setup.py build
+
+.PHONY: dist
+dist:
+	python setup.py sdist bdist_wheel
+
+.PHONY: install
+install:
+	pip install .
+
+.PHONY: dev-mode
+dev-mode:
+	pip install --editable .

apps/stix-shifter/README.md

@@ -0,0 +1,84 @@
+STIX-Shifter Threat Bus
+=======================
+
+This app bridges the gap between Threat Bus and various security tools by
+leveraging
+[STIX-Shifter](https://github.com/opencybersecurityalliance/stix-shifter).
+
+STIX-Shifter is a tool and library to transform STIX patterns into native
+queries for a variety of (mostly commercial) security tools, like
+[IBM QRadar](https://www.ibm.com/security/security-intelligence/qradar) or
+[Splunk](https://www.splunk.com/). This app connects STIX-Shifter with Threat
+Bus and provides a simple way to communicate with the commercial tools of your
+choice via Threat Bus.
+
+## How It Works
+
+The `stix-shifter-threatbus` app uses ZeroMQ to connect with Threat Bus. To
+connect via ZeroMQ, users must first install and configure the
+[`threatbus-zmq-app`](https://pypi.org/project/threatbus-zmq-app/) plugin on
+their Threat Bus host.
+
+This app functions as middleman between Threat Bus and security tools supported
+by STIX-Shifter. It subscribes to indicator updates from the bus and uses
+STIX-Shifter to actively translate STIX-2 intelligence to native queries.
+The app then executes these queries via STIX-Shifter. [Result processing
+is yet to be implemented.]
+
+## Quick Start
+
+You can configure the app via a YAML configuration file. See
+`config.yaml.example` for an example config file.
+
+Install `stix-shifter-threatbus` in a virtualenv and start it by passing a
+config file:
+
+```sh
+python -m venv venv
+source venv/bin/activate
+make dev-mode
+stix-shifter-threatbus -c config.yaml
+```
+
+## Configuration
+
+Apart from the logging section, which is self-explanatory, users need to
+configure the `threatbus` endpoint of the ZerMQ-App plugin and an optional
+`snapshot` of historic threat intel data they want to fetch.
+
+Additionally, users must configure each STIX-Shifter module individually to use
+it with this app. You also must install the corresponding modules according to
+your configuration. For example, if you configure a key `splunk` in the
+`modules` section, you must install the `stix-shifter-modules-splunk`. Otherwise
+the app will throw an error. See below for an example:
+
+```
+threatbus: localhost:13370 # connect with Threat Bus via this endpoint
+snapshot: 300 # request 300 days of historic indicators
+modules:
+  # for details on a module's options, please see https://github.com/opencybersecurityalliance/stix-shifter/blob/master/OVERVIEW.md#how-to-use
+  # to use the key `splunk` you must install `stix-shifter-modules-splunk`
+  # same goes for any other key, e.g., `elastic`, `qradar`, etc...
+  splunk:
+    max_results: 100 # limit the number of events queried by STIX-Shifter
+    # https://github.com/opencybersecurityalliance/stix-shifter/blob/master/OVERVIEW.md#connection
+    connection:
+      host: localhost
+      port: 8089 # Management port
+      selfSignedCert: false
+    # https://github.com/opencybersecurityalliance/stix-shifter/blob/master/OVERVIEW.md#configuration
+    transmission:
+      auth:
+        username: admin
+        password: admin123
+    # https://github.com/opencybersecurityalliance/stix-shifter/blob/master/OVERVIEW.md#translate
+    translation: # {<Any required options specific to the particular data source>}
+    # The data_source is a STIX-2 DataSource (e.g., an `identity`) and is used
+    # to create a STIX bundle with the queried results. You configure it here
+    # and only once for this module.
+    data_source:
+      type: identity
+      identity_class: events
+      name: Splunk
+      id: identity--629a6400-8817-4bcb-aee7-8c74fc57482c
+```

apps/stix-shifter/config.yaml.example

@@ -0,0 +1,35 @@
+logging:
+  console: true
+  console_verbosity: DEBUG
+  file: true
+  file_verbosity: DEBUG
+  filename: stix-shifter.log
+
+threatbus: localhost:13370
+snapshot: 30
+modules:
+  # for details on a module's options, please see https://github.com/opencybersecurityalliance/stix-shifter/blob/master/OVERVIEW.md#how-to-use
+  # to use the key `splunk` you must install `stix-shifter-modules-splunk`
+  # same goes for any other key, e.g., `elastic`, `qradar`, etc...
+  splunk:
+    max_results: 100 # limit the number of events queried by STIX-Shifter
+    # https://github.com/opencybersecurityalliance/stix-shifter/blob/master/OVERVIEW.md#connection
+    connection:
+      host: localhost
+      port: 8089 # Management port
+      selfSignedCert: false
+    # https://github.com/opencybersecurityalliance/stix-shifter/blob/master/OVERVIEW.md#configuration
+    transmission:
+      auth:
+        username: admin
+        password: admin123
+    # https://github.com/opencybersecurityalliance/stix-shifter/blob/master/OVERVIEW.md#translate
+    translation: # {<Any required options specific to the particular data source>}
+    # The data_source is a STIX-2 DataSource (e.g., an `identity`) and is used
+    # to create a STIX bundle with the queried results. You configure it here
+    # and only once for this module.
+    data_source:
+      type: identity
+      identity_class: events
+      name: Splunk
+      id: identity--629a6400-8817-4bcb-aee7-8c74fc57482c

apps/stix-shifter/setup.py

@@ -0,0 +1,58 @@
+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+
+from setuptools import setup
+
+with open("README.md", "r") as fh:
+    long_description = fh.read()
+
+setup(
+    author="Tenzir",
+    author_email="engineering@tenzir.com",
+    classifiers=[
+        # https://pypi.org/classifiers/
+        "Development Status :: 3 - Alpha",
+        "License :: OSI Approved :: BSD License",
+        "Operating System :: MacOS :: MacOS X",
+        "Operating System :: POSIX :: Linux",
+        "Topic :: Scientific/Engineering :: Information Analysis",
+        "Topic :: Scientific/Engineering :: Interface Engine/Protocol Translator",
+        "Topic :: Security",
+        "Topic :: Software Development :: Object Brokering",
+        "Topic :: System :: Distributed Computing",
+    ],
+    description="Bridges the gap between Threat Bus and STIX-Shifter",
+    entry_points={
+        "console_scripts": [
+            "stix-shifter-threatbus=stix_shifter_threatbus.shifter:main"
+        ]
+    },
+    include_package_data=True,
+    install_requires=[
+        "black >= 19.10b",
+        "coloredlogs >= 14.0",
+        "confuse",
+        "pyzmq >= 19",
+        "stix2 >= 2.1",
+        "stix-shifter >= 3.4.2",
+        "stix-shifter-utils >= 3.4.2",
+        "threatbus >= 2021.4.29",
+    ],
+    keywords=[
+        "open source",
+        "threatbus",
+        "Threat Bus",
+        "threat intelligence",
+        "TI",
+        "TI dissemination",
+    ],
+    license="BSD 3-clause",
+    long_description=long_description,
+    long_description_content_type="text/markdown",
+    name="stix-shifter-threatbus",
+    packages=["stix_shifter_threatbus"],
+    python_requires=">=3.7",
+    setup_requires=["setuptools", "wheel"],
+    url="https://github.com/tenzir/threatbus",
+    version="2021.05.27",
+)