The connector uses the CrowdSec API to collect IPs from a partial dump of CrowdSec CTI. The number of IPs you will get in the dump will depend on your subscription.
For each IP, an Ipv4-Addr or IPv6-Addr observable is created (or updated) and enriched. Enrichment depends on the configurations below.
Configuration parameters are provided using environment variables as described below. Some of them are placed directly in the docker-compose.yml since they are not expected to be modified by final users once that they have been defined by the developer of the connector.
| Docker environment variable | Mandatory | Type | Description |
|---|---|---|---|
OPENCTI_URL |
Yes | String | The URL of the OpenCTI platform. |
OPENCTI_TOKEN |
Yes | String | The default admin token configured in the OpenCTI platform parameters file. |
CONNECTOR_ID |
Yes | String | A valid arbitrary UUIDv4 that must be unique for this connector. |
CONNECTOR_NAME |
Yes | String | Name of the CrowdSec import connector to be shown in OpenCTI. |
CONNECTOR_SCOPE |
Yes | String | Supported scopes: IPv4-Addr, IPv6-Addr |
CONNECTOR_CONFIDENCE_LEVEL |
Yes | Integer | The default confidence level (an integer between 0 and 100). |
CONNECTOR_UPDATE_EXISTING_DATA |
No | Boolean | Enable/disable update of existing data in database. Default: false |
CONNECTOR_LOG_LEVEL |
No | String | The log level for this connector, could be debug, info, warn or error (less verbose). Default: info |
CROWDSEC_KEY |
Yes | String | CrowdSec CTI API key. See instructions to obtain it |
CROWDSEC_API_VERSION |
No | String | CrowdSec API version. Supported version: v2. Default: v2. |
CROWDSEC_DUMP_LISTS |
No | String | List of dump types separated by comma. - fire: contains the IPs part of the CrowdSec community blocklist. The number of IPs included is not fixed and can vary over time.- smoke: This dump contains most of the IPs reported by the CrowdSec community. This dump includes a fixed number of IPs according to your subscription. We propose a dump of the smoke database for the top 250K, 500K, 1M or a custom amount of IPs.Default: 'fire'. |
CROWDSEC_IMPORT_INTERVAL |
No | Number | Interval in hours between two imports. Default: 24 |
CROWDSEC_ENRICHMENT_THRESHOLD_PER_IMPORT |
No | Number | Maximum number of IP addresses to enrich in one import. Default: 10000 |
CROWDSEC_MAX_TLP |
No | String | Do not send any data to CrowdSec if the TLP of the observable is greater than crowdsec_max_tlp. Default: TLP:AMBER |
CROWDSEC_TLP |
No | String | TLP for created observable. Possible values are: TLP_WHITE, TLP_GREEN, TLP_AMBER, TLP_RED . If not set (None value), observable will be created without TLP. If an observable already exists, its TLP will be left unchanged.Default: None |
CROWDSEC_LABELS_SCENARIO_NAME |
No | Boolean | Enable/disable labels creation based on CTI scenario's name. Default: true |
CROWDSEC_LABELS_SCENARIO_LABEL |
No | Boolean | Enable/disable labels creation based on CTI scenario's label. Default: false |
CROWDSEC_LABELS_SCENARIO_COLOR |
No | String | Color of scenario based labels. Default: #2E2A14  |
CROWDSEC_LABELS_CVE |
No | Boolean | Enable/Disable CTI cve name based labels. Default: true |
CROWDSEC_LABELS_CVE_COLOR |
No | String | Color of cve based labels. Default: #800080  |
CROWDSEC_LABELS_MITRE |
No | Boolean | Enable/Disable CTI mitre technique based labels. Default: true |
CROWDSEC_LABELS_MITRE_COLOR |
No | String | Color of mitre technique based labels. Default: #000080  |
CROWDSEC_LABELS_BEHAVIOR |
No | Boolean | Enable/Disable CTI behavior based labels. Default: false |
CROWDSEC_LABELS_BEHAVIOR_COLOR |
No | String | Color of behavior based labels. Default: #808000  |
CROWDSEC_LABELS_REPUTATION |
No | Boolean | Enable/Disable CTI reputation based labels. Default: true |
CROWDSEC_LABELS_REPUTATION_MALICIOUS_COLOR |
No | String | Color of malicious reputation label. Default: #FF0000  |
CROWDSEC_LABELS_REPUTATION_SUSPICIOUS_COLOR |
No | String | Color of suspicious reputation label. Default: #FFA500  |
CROWDSEC_LABELS_REPUTATION_SAFE_COLOR |
No | String | Color of safe reputation label. Default: #00BFFF  |
CROWDSEC_LABELS_REPUTATION_KNOWN_COLOR |
No | String | Color of safe reputation label. Default: #808080  |
CROWDSEC_INDICATOR_CREATE_FROM |
No | String | List of reputations to create indicators from (malicious, suspicious, known, safe) separated by comma. Default: empty 'malicious,suspicious,known'.If an IP is detected with a reputation that belongs to this list, an indicator based on the observable will be created. |
CROWDSEC_ATTACK_PATTERN_CREATE_FROM_MITRE |
No | Boolean | Create attack patterns from MITRE techniques If an indicator has been created, there will be a targets relationship between the attack pattern and the indicator. Otherwise, there will be a related-to relationship between the attack pattern and the observable There will be a targets relationship between the attack pattern and a location created from targeted country.Default true |
CROWDSEC_VULNERABILITY_CREATE_FROM_CVE |
No | Boolean | Create vulnerability from CVE. There will be a related-to relationship between the vulnerabilty and the observableDefault true |
CROWDSEC_CREATE_NOTE |
No | Boolean | Enable/disable creation of a note in observable for each enrichment. Default: true |
CROWDSEC_CREATE_SIGHTING |
No | Boolean | Enable/disable creation of a sighting of observable related to CrowdSec organization. Default: true |
CROWDSEC_LAST_ENRICHMENT_DATE_IN_DESCRIPTION |
No | Boolean | Enable/disable saving the last CrowdSec enrichment date in observable description. Default: true |
CROWDSEC_MIN_DELAY_BETWEEN_ENRICHMENTS |
No | Number | Minimum delay (in seconds) between two CrowdSec enrichments. Default: 86400 Use it to avoid too frequent calls to CrowdSec's CTI API. Requires the last CrowdSec enrichment to be saved in the description, as we'll be comparing this date with the current one. if you are also using the CrowdSec Internal Enrichment connector, please ensure to also set CROWDSEC_LAST_ENRICHMENT_DATE_IN_DESCRIPTION=true and a sufficiently high value of CROWDSEC_MIN_DELAY_BETWEEN_ENRICHMENTS in the internal enrichment connector. |
CROWDSEC_CREATE_TARGETED_COUNTRIES_SIGHTINGS |
No | Boolean | Enable/Disable creation of a sighting of observable related to a targeted country Default: falseSighting count represents the percentage distribution of the targeted country among all the countries targeted by the attacker. |
You could also use the config.ymlfile of the connector to set the variable.
In this case, please put the variable name in lower case and separate it into 2 parts using the first underscore _. For example, the docker setting CROWDSEC_MAX_TLP=TLP:AMBER becomes :
crowdsec:
max_tlp: 'TLP:AMBER'You will find a config.yml.sample file as example.
In this example, we chose 146.70.186.190 as it is currently reported for cve and mitre techniques.
The result of a CrowdSec's enrichment should be similar to the following description:
- With regard to the observable itself, you should see:
- a list of dark olive green scenario name labels (
crowdsecurity/http-admin-interface-probing,crowdsecurity/http-bad-user-agent, etc.) - a list of purple cve labels (
cve-2021-41773, etc.) - a red
maliciousreputation label - An external reference to the CrowdSec CTI's url
- A note with some content (confidence, first seen, last seen, behaviors, targeted countries, etc.)
- A list of relationships:
relatedrelationships leading to vulnerabilities created from CVEsbased-onrelationship leading to a CrowdSec CTI indicator
- A sighting related to CrowdSec with the first and last seen information
- a list of dark olive green scenario name labels (
- As the
CROWDSEC_INDICATOR_CREATE_FROMrecommended setting containsmaliciousreputation, an indicator has been created with:- An external reference to the blocking list from which the flagged IP originates.
- A list of
indicatesrelationship leading to attack patterns created using mitre techniques- If you follow one of this relationship, you can navigate to the attack pattern created, where you will see
- An external reference to the MITRE ATT&CK url
- A list of
targetsrelationships leading to location created from targeted countries (Canada,Poland, etc.)
- If you follow one of this relationship, you can navigate to the attack pattern created, where you will see
An API key is limited to 24 queries per day. This should be taken into account when setting the import job frequency (see above CROWDSEC_IMPORT_INTERVAL configuration ).
As mentioned in this blog article, it's really hard tosay how long it will take to ingest thousands of IPs. The most honest answer is :
Well, it depends...
We carried out benchmarking tests on 2 separate servers (Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz), using the OpenCTI 6.1 platform.
- Server 1 with 8GB ram and 2 cores: we have allocated 2GB for Elastic Search
- Server 2 with 32Gb ram and 8 cores: we have allocated 16GB for Elastic Search
We used the default docker-compose.yml file provided by OpenCTI. In particular, we have left 3 workers in both cases.
The benchmarks were performed on a fresh installation, with no other connectors running (other than the default ones).
We used two types of import configuration:
- a LIGHT import with the poorest possible enrichment: create only an observable with an external reference pointing to the CrowdSec CTI url:
- CROWDSEC_LABELS_REPUTATION=false
- CROWDSEC_LABELS_SCENARIO_NAME=false
- CROWDSEC_LABELS_SCENARIO_LABEL=false
- CROWDSEC_LABELS_CVE=false
- CROWDSEC_LABELS_MITRE=false
- CROWDSEC_LABELS_BEHAVIOR=false
- CROWDSEC_INDICATOR_CREATE_FROM=''
- CROWDSEC_ATTACK_PATTERN_CREATE_FROM_MITRE=false
- CROWDSEC_VULNERABILITY_CREATE_FROM_CVE=false
- CROWDSEC_CREATE_NOTE=false
- CROWDSEC_CREATE_TARGETED_COUNTRIES_SIGHTINGS=false
- CROWDSEC_CREATE_SIGHTING=false- a FULL import with the richest possible enrichment: all labels, objects and relationships:
- CROWDSEC_LABELS_REPUTATION=true
- CROWDSEC_LABELS_SCENARIO_NAME=true
- CROWDSEC_LABELS_SCENARIO_LABEL=true
- CROWDSEC_LABELS_CVE=true
- CROWDSEC_LABELS_MITRE=true
- CROWDSEC_LABELS_BEHAVIOR=true
- CROWDSEC_INDICATOR_CREATE_FROM='malicious,suspicious,known'
- CROWDSEC_ATTACK_PATTERN_CREATE_FROM_MITRE=true
- CROWDSEC_VULNERABILITY_CREATE_FROM_CVE=true
- CROWDSEC_CREATE_NOTE=true
- CROWDSEC_CREATE_TARGETED_COUNTRIES_SIGHTINGS=true
- CROWDSEC_CREATE_SIGHTING=true We analyzed 3 metrics:
- the CrowdSec Python process time: the time required by the connector's Python process to handle a given number of IPs. It measures the time needed to retrieve the CrowdSec dump, analyze all the IPs, format all the available data to enrich an observable and send all the necessary bundle to the OpenCTI workers.
- the Total time for ingestion: the total time to ingest all IP. This this the time elapsed between the start and end of the import.
- the Average number of bundles ingested per seconds
We have obtained the following benchmarks.
To compare a light import and a full import, we used the Server 1 for 2000 IP addresses.
| Light Import | Full Import | |
|---|---|---|
| CrowdSec Python process time | 90s | 380s |
| Number of bundles sent | 2000 | 39119 |
| Total time for ingestion | 393s (6m33s) | 7050s (2h25m30s) |
| Average number of bundles ingested per seconds | 5.08 | 5.55 |
We can see that, depending on enrichment quality, import time varies by a factor of 1 to 18.
To compare Server 1 and Server 2 performances, we used a full import of 2000 IP addresses.
As the IP addresses retrieved and the associated CTI data vary from import to import, the number of bundles sent also varies.
| Server 1 | Server 2 | |
|---|---|---|
| CrowdSec Python process time | 380s | 289s |
| Number of bundles sent | 39119 | 38980 |
| Total time for ingestion | 7050s (2h25m30s) | 4925s (1h22m5s) |
| Average number of bundles ingested per seconds | 5.55 | 7,91 |
We can see that, depending on server configuration, import time varies by a factor of 1 to 1,4
We also compared the time needed to ingest different numbers of IP addressses using Server 2 and a full import configuration.
| 2000 IPs | 10000 IPs | 50000 IPs | |
|---|---|---|---|
| CrowdSec Python process time | 289s | 1436s (23m56s) | 2236s (1h36m6s) |
| Number of bundles sent | 38980 | 195519 | 971287 |
| Total time for ingestion | 4925s (1h22m5s) | 30346s (8h25m46s) | 305125s (3d12h45m25s) |
| Average number of bundles ingested per seconds | 7,91 | 6,44 | 3,18 |
We can see that, with the current server configuration, the more IP addresses we ingest, the lower the average number of bundles ingested per second.
As the time of writing, there are ongoing issues on the OpenCTI GitHub repository. We'll continue to monitor them to see what could be done to improve ingestion performance.