Support for deployment scripts

[b-meson]

We have been getting a lot of requests for deployment support. We should add support for the following, maybe in Aansible.

The install script should at minimum run the following

• Install an nginx user and install the application running as the user (non-root)
• Expose nginx as a reverse proxy for a gunicorn app to connect to
• Install Let's Encrypt and install the TLS certificates.
• Deploy OO and connect the app to the reverse proxy.
• Install a basic firewall (ufw would be fine) and remove any ports that aren't HTTPS and SSH.
• Prompt the user for the .env variables and write them to a file.
• Extra points for an upgrade or backup role from Amazon's S3.

[fritzdavenport]

Is there anyone I can pair with @b-meson - if I can make a docker-compose or ansible playbook, to test it going to a fresh box in production?

[b-meson]

@fritzdavenport absolutely! I can create as many testing / staging servers as you need in our cloud environment. I think there is one more thing to understand: do we want to create something like a k8 and use our containers in prod, or do we instead want to run the applications on bare metal (as we currently do, but not everyone is doing it this way)

[fritzdavenport]

Had a larger conversation about this in the monthly meeting. Want to outline a few possibilities in this issue's discussion to generally guide the conversation and what the group wants to do. We currently can or do support three different deployment methods - being 1) git+ssh, 2) ansible, 3) docker-compose. I want to outline the three approaches, hopefully consolidate to one (or two, one of which is used only for testing), and define what we are willing to support in terms of documentation and out-of-the-box deployments for forked copies of the repo

[fritzdavenport]

Git + SSH (Existing Solution)

Probably the easiest solution in terms of infrastructure investment. A frequent combination for many developers and projects. A lightweight and common combination, but difficult to piece together after a certain point and potentially less-stable.

Unique Pros

• Simplest solution in terms of technology
• common - bash/ssh/git is ubiquitous, but requires knowledge
• smallest deployment footprint for production host

Unique Cons

• SSH Key management (for both ssh'ing onto the box and pulling from git)
• (drift) no dependency management for production host
• • difficult to manage required software / dependencies
• no ability to reconcile to a known state without manual intervention
• (insecure) need to get into a priviledged shell to intervene

Unique Client Requirements

Unique Server Requirements

• git

Testing Demo

• Our current setup offloads to docker-compose to have a repeatable environment for testing, git+ssh doesn't offer any tooling to help local testing.
• git+ssh can be used to provision our staging environment with CI/CD

Deployment Demo

• https://github.com/lucyparsons/OpenOversight/blob/develop/fabfile.py#L51-L57

[fritzdavenport]

Ansible

Ansible is an established devops tool that primarily uses python over ssh to provision a host
https://github.com/lucyparsons/OpenOversight-ansible

Unique Pros

• builds on top of git+ssh and a structured approach like the fabfile, groups tasks using YAML
• provisions underlying host
• lots of community support for premade playbooks and roles to install other things (firewalls, databases, fail2ban) https://galaxy.ansible.com/
• includes user and system management
• good community, well supported

Unique Cons

• playbooks are generally built for a specific operating system (ubuntu vs centos)
• drift can still occur if playbooks are not run regularly, and folks have shell access

Unique Client Requirements

• ansible

Unique Server Requirements

Testing Demo

• Our current setup offloads to docker-compose to have a repeatable environment for testing
• ansible can be used to provision our staging environment with CI/CD

Deployment Demo

• https://github.com/lucyparsons/OpenOversight-ansible/blob/main/roles/openoversight/tasks/application.yml (note: from @dismantl and bpdwatch.com )

- name: Create OpenOversight directory
  become: true
  file:
    state: directory
    path: "{{ environ.path }}"
    owner: "{{ openoversight_user }}"
    group: "{{ openoversight_user }}"

- name: Checkout OpenOversight git repo
  git:
    repo: "{{ openoversight_git_repo }}"
    dest: "{{ environ.path }}/OpenOversight"
    version: "{{ environ.git_version }}"
    update: yes
...

[fritzdavenport]

Docker

Docker is a lightweight virtualization layer which allows process isolation and a solid API for deploying pre-built software bundles.

For reference: We are not looking to pursue Kubernetes or K3s - we do not want to support deployment and management of that orchestration platform, and the additional benefits are probably unneeded for our use cases (single deploys to single hosts). If there's enough interest for kubernetes manifests, they can be provided for those users. We may pursue docker-compose on a single host instead.

Unique Pros

• unified testing, staging, production environments
• easy rollback / forward, simple deploys
• supported on every system with docker, base system is unimportant
• sandboxed - more secure in face of some threats
• lots of software is already dockerized and available out-of-the-box at hub.docker.com
• good community, well supported

Unique Cons

• debugging requires specialized knowledge (easier, to just push a new deploy than debug a live image)
• harder to get a shell for manual intervention (though, is an anti-pattern)
• requires an image registry (quay.io / dockerhub)
• requires (rather small) installation of docker daemon to host or a docker-only VMI

Unique Client Requirements

• docker

Unique Server Requirements

• docker

Testing Demo

Deployment Demo

[dismantl]

For bpdwatch.com we've used Ansible to deploy docker-compose.yml files that are fully customized with all the relevant environment variables, so it has the benefits of both the Docker and Ansible options. I've found it makes administration pretty easy; no worries about dependency management or the host environment. I put that in https://github.com/lucyparsons/OpenOversight-ansible, but instead of deploying docker-compose.yml files it just manages the Docker containers and image directly since I thought that would be simpler and more direct.

Also while Nginx is great, I went for Traefik instead since we were deploying via Docker. It's really great since proxied endpoints are managed dynamically via Docker container labels so the process is pretty seamless. Plus it handles all the SSL via Let's Encrypt.

[msaad7777]

It's great to see that there are multiple options for deploying the application. Using Docker and Ansible can offer several benefits, such as easier dependency management, increased stability, and the ability to manage multiple environments more efficiently. Additionally, using Traefik as a reverse proxy can simplify the process of managing proxied endpoints and SSL with Let's Encrypt.

Regarding the issue at hand, I would be happy to help tackle this task. In particular, I could work on the following tasks: installing an nginx user and setting it up to run the application as a non-root user, configuring the reverse proxy to connect to the gunicorn app, installing Let's Encrypt and setting up the TLS certificates, deploying OO and connecting it to the reverse proxy, setting up a basic firewall and removing any unnecessary ports, and prompting the user for the necessary .env variables and writing them to a file. Additionally, I could work on implementing an upgrade or backup role from Amazon's S3 for bonus points.

Please let me know if you would like me to take on this issue, and if there are any additional details or requirements that I should be aware of.

[abandoned-prototype]

Hi @msaad7777, welcome and apologies for the late response! Help with this issue would be greatly appreciated.
The steps you laid out make a lot of sense to me, this is very similar to what I just recently implemented, without using Ansible however. So having this put into code and therefore easily repeatable would be a great help to anyone trying to deploy a new OpenOversight server. Thanks for your willingness to help out! Please let us know here, if you run into any issues