Merge pull request #103 from strongdm/feat/support-prometheus-monitoring

Support prometheus monitoring

.ruby-version

@@ -0,0 +1 @@
+2.7.1

Gemfile

@@ -20,4 +20,5 @@ gem 'fluent-plugin-grafana-loki'
 gem 'fluent-plugin-remote_syslog'
 gem 'fluent-plugin-elasticsearch'
 gem 'fluent-plugin-bigquery'
+gem 'fluent-plugin-prometheus'
 gem 'test-unit'

Gemfile.lock

@@ -119,6 +119,9 @@ GEM
     fluent-plugin-mongo (1.5.0)
       fluentd (>= 0.14.22, < 2)
       mongo (~> 2.6.0)
+    fluent-plugin-prometheus (2.0.3)
+      fluentd (>= 1.9.1, < 2)
+      prometheus-client (>= 2.1.0)
     fluent-plugin-remote_syslog (1.0.0)
       fluentd
       remote_syslog_sender (>= 1.1.1)
@@ -341,6 +344,7 @@ DEPENDENCIES
   fluent-plugin-kafka
   fluent-plugin-logzio
   fluent-plugin-mongo
+  fluent-plugin-prometheus
   fluent-plugin-remote_syslog
   fluent-plugin-rewrite-tag-filter
   fluent-plugin-s3

README.md

@@ -18,10 +18,11 @@ The Log Export Container is a Docker Image you can use for spinning up multiple
 
 1. Download the `docker-compose.yml` file from the Github repo onto your machine (or copy-paste its contents into a file you created directly on the machine with the same name).
    - Make sure that the 'Required variables' in the .yml file are set appropriately based on your desired log format and output destination.
-2. Run `sudo docker-compose up`
+2. Run with your preferred container orchestrator (with docker, you can simply run `docker-compose up`)
 3. Log into the strongDM Admin UI and go to the Settings page, then the Log Encryption & Storage tab.
 4. Set "Log locally on relays?" to 'Yes'
-5. Set "Local storage?" to "Syslog" and enter the IP address of the machine running the Log Export Container along with port 5140 ![image](https://user-images.githubusercontent.com/7840034/127934335-239b5e97-772c-4ac6-8e66-864ffaf4cccc.png)
+5. Set "Local storage?" to "Syslog" and enter the IP address of the machine running the Log Export Container along with port 5140
+   - ![image](https://user-images.githubusercontent.com/7840034/127934335-239b5e97-772c-4ac6-8e66-864ffaf4cccc.png)
    - Make sure that port 5140 on the machine hosting the container is accesible from your gateways. You can also host the container on your gateways themselves.
 6. Set "Local format?" to match the input format you specified in the .yml file.
 7. Click "Update" and you're done!
@@ -39,7 +40,10 @@ If you want to modify the container and quickly see the changes in your local, y
 ./dev-tools/start-container.sh
 ```
 
-You could also run the project in your local without docker, please refer to [CONFIGURE_LOCAL_ENV](docs/CONFIGURE_LOCAL_ENV.md)
+You could also run the project in your local without docker, please refer to [CONFIGURE_LOCAL_ENV](docs/deploy_log_export_container/CONFIGURE_LOCAL_ENV.md)
+
+## Monitoring
+Currently the application supports Prometheus Metrics about the received and forwarded logs. For more details, please see [CONFIGURE_PROMETHEUS](docs/monitoring/CONFIGURE_PROMETHEUS.md)
 
 ## Contributing
 Refer to the [contributing](CONTRIBUTING.md) guidelines or dump part of the information here.

conf-utils.rb

@@ -32,6 +32,13 @@ def extract_activity_interval
   interval
 end
 
+def monitoring_conf
+  monitoring_enabled = extract_value(ENV['LOG_EXPORT_CONTAINER_ENABLE_MONITORING']) == "true"
+  if monitoring_enabled
+    File.read("#{ETC_DIR}/monitoring.conf")
+  end
+end
+
 def output_stores_conf
   conf = ""
   output_types = extract_value(ENV['LOG_EXPORT_CONTAINER_OUTPUT'])

create-conf.rb

@@ -7,6 +7,7 @@ def create_file
   File.open("#{ETC_DIR}/fluent.conf", "w") do |f|
     f.write(input_conf)
     f.write(input_extract_audit_activities_conf)
+    f.write(monitoring_conf)
     f.write(input_extract_audit_entity_conf("resources"))
     f.write(input_extract_audit_entity_conf("users"))
     f.write(input_extract_audit_entity_conf("roles"))

dev-tools/grafana/Dockerfile

@@ -0,0 +1,16 @@
+FROM grafana/grafana
+
+ENV GF_AUTH_DISABLE_LOGIN_FORM="true"
+ENV GF_AUTH_ANONYMOUS_ENABLED="true"
+ENV GF_AUTH_ANONYMOUS_ORG_ROLE="Admin"
+
+COPY grafana.ini /etc/grafana/
+COPY init_dashboard.sql /
+
+USER root
+RUN apk add sqlite
+
+RUN sqlite3 /var/lib/grafana/grafana.db < /init_dashboard.sql
+RUN rm /init_dashboard.sql
+
+USER grafana