Malcolm v3.3.1 development (cisagov#174)

Minor Malcolm release with the following updates:

* Bump capa to [v3.0.2](https://github.com/mandiant/capa/releases/tag/v3.0.2) which now includes ELF scanning capabilities
* Bump zeek to [v4.0.4](https://github.com/zeek/zeek/releases/tag/v4.0.4)
* Incorporate Corelight's ["OMIGOD" (CVE-2021-38647)](https://github.com/corelight/CVE-2021-38647) plugin
* minor fix on race condition creating default anomaly detectors
* minor tweak to `build.sh` script for building docker images

3.3.0 development (cisagov#173)

* New features
    * Automatically create some broadly useful anomaly detectors when initializing Kibana
        * connection size
        * file transfer MIME type
        * action and result (by application protocol)
    * Configurable [event severity scoring](https://github.com/cisagov/malcolm/tree/main#Severity) (idaholab#19) and new **Severity** dashboard

* Other changes
    * vagrant-based ISO build can now work with either VirtualBox or libvirt providers
    * change wording of terms such as "master"/"slave" to "client"/"server" as instructed by DHS directive

* Version updates
    * Update base image for Debian-based Docker images from 10 (buster) to 11 (bullseye)
    * Update Yara to 4.1.2
    * Update Capa to 2.0.0
    * Update Spicy to 1.2.1
    * Update remainder of python 2 code to python 3

address idaholab#52, LDAP parser broken (ldap.spicy:393 unset optiona…

…l value) if built from source since May 31, by reverting one upstream commit in my spicy-analyzers fork

v3.2.0 fixes (v3.1.1 had a few regressions, otherwise this is the sam…

…e release as that)

fix idaholab#51, kibana offline maps server not started
fix idaholab#50, zeek_template index template not created if index management not configured

3.1.1 development (cisagov#172)

v3.1.1 development

* New features
  * ["Best Guess" Fingerprinting for ICS Protocols](idaholab#49) - In an effort to help identify more ICS traffic, Malcolm can use "buest guess" method based on transport protocol (e.g., TCP or UDP) and port(s) to categorize potential traffic communicating over some ICS protocols without full parser support. This feature involves a [mapping table](https://github.com/idaholab/Malcolm/blob/master/zeek/config/guess_ics_map.txt) and a [Zeek script](https://github.com/idaholab/Malcolm/blob/master/zeek/config/guess.zeek) to look up the transport protocol and destination and/or source port to make a best guess at whether a connection belongs to one of those protocols. These potential ICS communications are categorized by vendor where possible. The list of ICS protocols' ports was adapted from various public sources, including, but not limited to, [Grassmarlin](https://github.com/nsacyber/GRASSMARLIN/tree/master/GM3/data/fingerprint)'s fingerprints and [ITI/ICS-Security-Tools](https://github.com/ITI/ICS-Security-Tools/blob/master/protocols/PORTS.md)' list of Control Systems Ports.

* Improvements and bug fixes
  * Allow configuring the number of concurrent requests for ClamAV scanning, Yara and Capa via environment variables (`CLAMD_MAX_REQUESTS`, `YARA_MAX_REQUESTS` and `CAPA_MAX_REQUESTS`)
  * Zeek plugins to detect [CVE-2021-31166](https://github.com/corelight/CVE-2021-31166) and [pingback](https://github.com/corelight/pingback) vulnerabilities
  * Move creation of custom fields and views to [Arkime's config.ini](https://arkime.com/settings#custom-fields)
  * LDAP bind credentials world readable in docker (idaholab#47 and cisagov#171)
  * Deny access to uploaded files (cisagov#170)

* Version bumps
  * Yara to 4.1.1
  * Zeek to 4.0.3
  * Spicy to 1.1.0
  * Alpine to 3.14
  * NGINX to 1.20.1
  * Linux kernel to 5.10 (for ISO installs)
  * urllib3 to 1.26.5 (cisagov#169)

Malcolm v3.1.0 development (cisagov#165)

* [Network analyzers](https://github.com/cisagov/malcolm#Protocols)
    - Added support for [EtherCAT](https://en.wikipedia.org/wiki/EtherCAT) ([ICS protocol](https://github.com/cisagov/icsnpp-ethercat))
    - Fixed and improved Spicy-based [LDAP analyzer](zeek/spicy-analyzers#56)
    - Detect VPN [protocols](https://github.com/zeek/spicy-analyzers/tree/main/analyzer/protocol) IPsec, OpenVPN and WireGuard

* New or improved
    - Updated many Kibana dashboards and added dashbaords for newly-supported network protocols
    - Improved output of debug logs from docker images
    - Many minor improvements to underlying system for ISO installations
    - **Massively** cut build time for Hedgehog ISO and Zeek Docker container by using .deb packages from released versions rather than building from source
    - During build, [install all Zeek plugins](https://github.com/cisagov/Malcolm/blob/master/shared/bin/zeek_install_plugins.sh) via zkg

* Version updates
    - **[Zeek](https://github.com/zeek/zeek/releases) v4.0.1**
    - [Spicy](https://github.com/zeek/spicy) v1.0.0
    - [Open Distro For Elasticsearch](https://opendistro.github.io/for-elasticsearch-docs/version-history/) v1.13.2
    - [Yara](https://github.com/VirusTotal/yara/releases) v4.1.0
    - [Capa](https://github.com/fireeye/capa/releases) v1.6.3
    - switch from centos:7 to [amazonlinux:2](https://hub.docker.com/_/amazonlinux) for base Docker image to build Kibana plugins
    - [stunnel](https://www.stunnel.org/NEWS.html) v5.59
    - [NGINX](https://nginx.org/) v1.20.0
    - [LLVM/clang](https://releases.llvm.org/11.0.1/docs/ReleaseNotes.html) toolchain v11
    - Flask-Cors v3.0.9 for Hedgehog kiosk interface (dependabot-flagged [security alert](https://nvd.nist.gov/vuln/detail/CVE-2020-25032))
    - latest updates of various Zeek plugins, system and python packages, etc.
    - all Python scripts updated to Python 3

* Bugs fixed
    - When LDAP authentication is used instead of BASIC authentication, show a landing page rather than a server error when attempting to browse to the local authentication management interface
    - Fixed a [regression bug](idaholab#42) where Malcolm fails to start correctly if not using UID/GID 1000:1000
    - [Don't automatically expose](idaholab#38) elasticsearch (and logstash) ports unless explicitly configured to do so
    - freshclam should update the clamav database [during docker image build](idaholab#39)

v3.0.1 development (cisagov#161)

Malcolm v3.0.1

- Version bumps
  - Open Distro for Elastic ([v1.13.0](https://github.com/opendistro-for-elasticsearch/opendistro-build/blob/main/release-notes/opendistro-for-elasticsearch-release-notes-1.13.0.md)), which adds the following functionality over the previous release
    - [Reporting](https://opendistro.github.io/for-elasticsearch-docs/docs/kibana/reporting/)
    - [Historical data anomaly detection](https://opendistro.github.io/for-elasticsearch-docs/docs/ad/#step-6-analyze-historical-data)
  - ODFE v1.13.0 is based on the Elastic components 7.10.2 ([elasticsearch](https://www.elastic.co/guide/en/elasticsearch/reference/current/release-notes-7.10.2.html), [kibana](https://www.elastic.co/guide/en/kibana/current/release-notes-7.10.2.html), [logstash](https://www.elastic.co/guide/en/logstash/current/logstash-7-10-2.html), [beats](https://www.elastic.co/guide/en/beats/libbeat/master/release-notes-7.10.2.html))
  - Zeek [3.0.13](https://github.com/zeek/zeek/releases/tag/v3.0.13)
  - NGINX [1.19.7](https://nginx.org/en/CHANGES)
  - Alpine Linux [3.13](https://alpinelinux.org/posts/Alpine-3.13.0-released.html) Docker base layer
  - docker-compose [1.28.5](https://docs.docker.com/compose/release-notes/) in Malcolm installable ISO version
- Restored the [sankey visualization](https://github.com/uniberg/kbn_sankey_vis) which was temporarily removed in Malcolm v3.0.0 (although there are still a few minor cosmetic [issues](uniberg/kbn_sankey_vis#15) with it)
- Removed port 8443 for upload (now just use /upload over the regular HTTPS port)
- Fixed issue with ODFE email alerts not being able to use self-signed SMTP certificates by importing CA certs in `nginx/ca-trust` into the JDK trust store for Elasticsearch and Logstash (see idaholab#37)
- Don't expose the Elasticsearch 9200 by default, it must now be explicitly be enabled during `install.py -c` (see idaholab#38)
- For ISO-installed versions of Malcolm and Hedgehog Linux, populate `/etc/os-release` with information about the build/release version
- Populate user-agent for a few clients ([Arkime's moloch-capture](arkime/arkime#1615), some hedgehog test connection processes) so they're not just sent as blank when communicating with Malcolm
- Added Arkime link to Kibana dashboards' navigation pane
- Fix some issues in control script with older python3 versions (3.6.x) with `contextlib.nullcontext` not being available
- Fix suggestion for yum-based distributions to install python 3 requests via pip

List of changes in Malcolm v3.0.0:

- Change base for Elasticsearch and Kibana Docker images (version 7.6.2) from Elastic.co to Open Distro for Elastic (based on Elastic 7.10.0); see idaholab#15. This is a major change which **breaks backwards compatibility** for several features (listed below). If you are using these features, you will need to back up the data and/or configuration associated with them and migrate them manually to the new tools. No automatic migration or upgrade of these features is performed. It's recommended that you re-run `install.py --configure` (see [System configuration and tuning](https://github.com/cisagov/Malcolm#ConfigAndTuning)) prior to running Malcolm v3.0.0.
  - Kibana [comments](https://github.com/gwintzer/kibana-comments-app-plugin) replaced with [Notebooks](https://opendistro.github.io/for-elasticsearch-docs/docs/notebooks/)
  - Kibana [elastalert](https://github.com/nsano-rururu/elastalert-kibana-plugin) plugin replaced with [Alerting](https://opendistro.github.io/for-elasticsearch-docs/docs/alerting/) plugin
  - Elasticsearch [curator](https://www.elastic.co/guide/en/elasticsearch/client/curator/current/index.html) replaced with [Index Management](https://opendistro.github.io/for-elasticsearch-docs/docs/ism/) plugin
  - The third-party [Sankey visualization plugin](https://github.com/mmguero-dev/kbn_sankey_vis] has been temporarily removed due to compatibility issues, although it is planned to be reintegrated in a Malcolm point release in the near future (see uniberg/kbn_sankey_vis#15)
  - The third-party [Kibana drill-down plugin](https://github.com/mmguero-dev/kibana-plugin-drilldownmenu/) providing Kibana-to-Moloch pivoting has been temporarily removed due to compatibility issues, although it is planned to be reintegrated in a Malcolm point release in the near future (see goodlabs-studio/kibana-plugin-drilldownmenu#5)
  - In addition to those replacements, the Real Time Anomaly Detection feature is now available:
    - [Real Time Anomaly Detection in Open Distro for Elasticsearch](https://opendistro.github.io/for-elasticsearch/blog/odfe-updates/2019/11/real-time-anomaly-detection-in-open-distro-for-elasticsearch/) blog announcement
     - [Anomaly Detection](https://opendistro.github.io/for-elasticsearch-docs/docs/ad/) documentation and source code for [Elasticsearch](https://github.com/opendistro-for-elasticsearch/anomaly-detection) and [Kibana](https://github.com/opendistro-for-elasticsearch/anomaly-detection-kibana-plugin) components
     - [Random Cut Forests writeup](https://opendistro.github.io/for-elasticsearch/blog/odfe-updates/2019/11/random-cut-forests/)

- Malcolm startup time (especially the Logstash container) has been reduced drastically

- Improvements to Malcolm's prebuilt Kibana dashboards

- Improvements to build scripts

- Minor tweaks and bugfixes for ISO-installed environments for Malcolm and Hedgehog Linux

- Minor other bug fixes and performance improvements

- Version bump
  - Yara [v4.0.5](https://github.com/VirusTotal/yara/releases/tag/v4.0.5)

Topic/2.6.1 merge (cisagov#159)

Malcolm v2.6.1 contains the following changes:

cisagov/Malcolm@v2.6.0...v2.6.1

* Added [TFTP](https://en.wikipedia.org/wiki/Trivial_File_Transfer_Protocol) [Zeek parser](https://github.com/zeek/spicy-tftp) and corresponding Logstash parsing, Arkime WISE support and Kibana dashboards
* Provide browser-based access to zeek/extracted-files directory (idaholab#34)
* Fix LDAP analyzer not parsing all events (idaholab#35)
* Provide more fine-tuned controls for Zeek's node.cfg in Hedgehog sensor (idaholab#36, cisagov/pull/158)
* set zeek.uid to conn_uids for files.log entries (idaholab#33)
* Modify Zeek build chain to use default GCC compilers instead of LLVM/clang,which reduces build dependencies
* Use Firefox instead of Chromium for browser in ISO-installed versions of Malcolm and in Hedgehog Linux
* Updated copyright notices in text from "2020" to "2021" (which is the bulk of the changed files in this commit)
* Version bumps
  * Yara to 4.0.4

changes for Release 2.6.0 (new ICSNPP Zeek parsers) (cisagov#157)