Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

kubernetes deployment - script consolidation #172

Closed
mmguero opened this issue Mar 29, 2023 · 1 comment
Closed

kubernetes deployment - script consolidation #172

mmguero opened this issue Mar 29, 2023 · 1 comment
Assignees
@mmguero
Labels
cloud Relating to deployment of Malcolm in the cloud and/or with Kubernetes install.py Relating to the install.py configuration script
Milestone
v23.05.0

Comments

@mmguero
Copy link
Collaborator

mmguero commented Mar 29, 2023

Sub-issue of #149

I don't think Malcolm's install.py script is going to install Kubernetes for anybody, but the install.py --configure part certainly could and probably should ask for relevant settings and/or skip settings that are docker-compose specific if they want to do a Kubernetes deployment. We've consolidated the environment variable configuration (see the config directory, control.py and install.py and the removal of these values from docker-compose.yml). We need to see how else to tweak the scripts to facilitate a Kubernetes deployment. I think minimally we could have the control.py script's start, stop, etc., reference call the appropriate kubectl --kubeconfig whatever.yaml (with the yaml file they provide) using either kubectl or the corresponding python library and start/stop/logs/status/etc. Malcolm that way. We've been using this testing script for starting and stopping Malcolm in Kubernetes, some of that will need to be adopted for creating configmaps, etc.
@mmguero mmguero added install.py Relating to the install.py configuration script cloud Relating to deployment of Malcolm in the cloud and/or with Kubernetes labels Mar 29, 2023
@mmguero mmguero mentioned this issue Apr 14, 2023
Closed
@mmguero mmguero self-assigned this Apr 14, 2023
mmguero added a commit that referenced this issue Apr 14, 2023
@mmguero
beginning work in progress on script consolidation for kubernetes, #172
1f9190e
mmguero added a commit that referenced this issue Apr 14, 2023
@mmguero
work in progress on script consolidation for kubernetes, #172
835c351
mmguero added a commit that referenced this issue Apr 14, 2023
@mmguero
work in progress on script consolidation for kubernetes, #172
a376bb5
mmguero added a commit that referenced this issue Apr 14, 2023
@mmguero
work in progress on script consolidation for kubernetes, #172
cceb085
mmguero added a commit that referenced this issue Apr 14, 2023
@mmguero
work in progress on script consolidation for kubernetes, #172
d7c461c
@mmguero
Copy link
Collaborator Author

mmguero commented Apr 14, 2023

A few notes as I've been working on this:

  • control.py (start, stop, restart, auth_setup, logs, status) and install.py (installation and configuration) will use the official kubernetes library for python for kubernetes mode
  • the user will specify a YML file via install.py -f/--configure-file or control.py -f/--file and it will automatically determine if we're talking about using docker-compose or kubernetes for orchestration based on the contents of the file

mmguero added a commit that referenced this issue Apr 17, 2023
@mmguero
work in progress on control.py's logs function for #172
b2aefab
mmguero added a commit that referenced this issue Apr 17, 2023
@mmguero
work in progress on control.py's function for #172
0e873f5
mmguero added a commit that referenced this issue Apr 17, 2023
@mmguero
work in progress on control.py's function for #172
3e2c208
mmguero added a commit that referenced this issue Apr 18, 2023
@mmguero
work in progress on script consolidation for kubernetes, #172
d2e1760
mmguero added a commit that referenced this issue Apr 18, 2023
@mmguero
work in progress on script consolidation for kubernetes, #172
731bc74
mmguero added a commit that referenced this issue Apr 18, 2023
@mmguero
work in progress on script consolidation for kubernetes, #172
716c91c
mmguero added a commit that referenced this issue Apr 18, 2023
@mmguero
work in progress on script consolidation for kubernetes, #172
aa04c8c
mmguero added a commit that referenced this issue Apr 18, 2023
@mmguero
work in progress on script consolidation for kubernetes, #172
728b55d
mmguero added a commit that referenced this issue Apr 18, 2023
@mmguero
work in progress on script consolidation for kubernetes, #172
a79483b
mmguero added a commit that referenced this issue Apr 18, 2023
@mmguero
work in progress on script consolidation for kubernetes, #172
4a315af
mmguero added a commit that referenced this issue Apr 18, 2023
@mmguero
work in progress on script consolidation for kubernetes, #172
1a5af9e
mmguero added a commit that referenced this issue Apr 18, 2023
@mmguero
work in progress on script consolidation for kubernetes, #172
6dc8da9
mmguero added a commit that referenced this issue Apr 18, 2023
@mmguero
work in progress on script consolidation for kubernetes, #172
d62fbaa
@mmguero mmguero closed this as completed Apr 18, 2023
mmguero added a commit that referenced this issue Apr 18, 2023
@mmguero
work in progress on script consolidation for kubernetes, #172
c633cd5
mmguero added a commit that referenced this issue Apr 18, 2023
@mmguero
work in progress on script consolidation for kubernetes, #172
2a935d4
mmguero added a commit that referenced this issue Apr 18, 2023
@mmguero
work in progress on script consolidation for kubernetes, #172
d964143
@mmguero mmguero mentioned this issue Apr 20, 2023
Open
mmguero added a commit that referenced this issue Apr 24, 2023
@mmguero
work for #172 script consolidation
c40976a
mmguero added a commit to mmguero-dev/Malcolm that referenced this issue Apr 25, 2023
@mmguero
work for idaholab#172 script consolidation
2fef7b3
@mmguero mmguero added this to the v23.05.0 milestone Apr 26, 2023
This was referenced May 1, 2023
Merged
Merged
mmguero added a commit that referenced this issue May 1, 2023
@mmguero
Merge pull request #191 from idaholab/v23.05.0_merge_idaholab
3438f9c 
Malcolm v23.05.0

Malcolm v23.05.0 is a major release with new features, enhancements, component version updates and bug fixes.

IMPORTANT NOTE: Malcolm v23.05.0 has completely changed the way it manages its settings: rather than using environment variables found at the top of the `docker-compose.yml` file, it uses environment variables in `.env` files inside of the `config` directory. The locations of a number of configuration files have also changed. It's not recommended to update to Malcolm v23.05.0 from a previous version of Malcolm. Instead, shut down Malcolm, rename your old Malcolm installation directory to something else, and reconfigure Malcolm using `./scripts/configure` and `./scripts/auth_setup`.

v23.04.0...v23.05.0

* New features
    - integrate [ICSNPP-Synchrophasor](https://github.com/cisagov/icsnpp-synchrophasor/) parser (#190)
    - [End-to-end Malcolm and Hedgehog Linux ISO Installation]((https://idaholab.github.io/Malcolm/docs/malcolm-hedgehog-e2e-iso-install.html#InstallationExample) document (#181)
    - support Malcolm deployment with Kubernetes (#149)
        + see [Deploying Malcolm with Kubernetes](https://idaholab.github.io/Malcolm/docs/kubernetes.html#Kubernetes)
        + This could be considered a "beta" release for Malcolm deployment with Kubernetes, as there is still [some work](https://github.com/idaholab/Malcolm/issues?q=is%3Aissue+is%3Aopen+kubernetes) to be done in this area. Please let us know what issues or suggestions you have via the [issue tracker](https://github.com/idaholab/Malcolm/issues) or via email to [malcolm@inl.gov](mailto:malcolm@inl.gov?subject=Malcolm).
        + contributing issues:
            * inotify issue (#168)
            * htadmin/nginx and htpasswd (#169)
            * opensearch (#170)
            * uploading large PCAP files (#171)
            * script consolidation (#172)
            * documentation (#173)
            * user-defined persistent volumes (#174)
            * opensearch keystore (#176)
            * expose other TCP services (#183)
            * provide with filebeat access to nginx access and error logs (#186)
            * use Secrets for some environment variables instead of ConfigMaps (#189)

* Enhancements and fixes
    * remove name-map-ui container (#165) in favor of using NetBox for asset identification
    * Python script refactoring, consolidation and cleanup
    * standardization of Docker container entrypoints
    * create `./scripts/configure` alias for `./scripts/install.py --configure`

* Component version updates
    - Arkime to [v4.3.0](https://github.com/arkime/arkime/blob/fa0db2415bdc109be7a4dd8ee2c2838673980b5f/CHANGELOG#L33-L72)
    - Capa to [v5.1.0](https://github.com/mandiant/capa/releases/tag/v5.1.0)
    - Fluent Bit to [v2.1.2](https://fluentbit.io/announcements/v2.1.2/)
    - NetBox to [v3.5.0](https://github.com/netbox-community/netbox/releases/tag/v3.5.0)
    - NGINX to [v1.22.1](http://nginx.org/en/CHANGES-1.22)
    - Supercronic to [v0.2.24](https://github.com/aptible/supercronic/releases/tag/v0.2.24)
    - Suricata to [v6.0.10](https://suricata.io/2023/01/31/suricata-6-0-10-released/)
    - Yara to [v4.3.0](https://github.com/VirusTotal/yara/releases/tag/v4.3.0)
    - Zeek to [v5.2.1](https://github.com/zeek/zeek/releases/tag/v5.2.1)
mmguero added a commit to cisagov/Malcolm that referenced this issue May 1, 2023
@mmguero
Merge pull request #249 from cisagov/v23.05.0_merge_cisagov
c02a6e9 
Malcolm v23.05.0

Malcolm v23.05.0 is a major release with new features, enhancements, component version updates and bug fixes.

IMPORTANT NOTE: Malcolm v23.05.0 has completely changed the way it manages its settings: rather than using environment variables found at the top of the `docker-compose.yml` file, it uses environment variables in `.env` files inside of the `config` directory. The locations of a number of configuration files have also changed. It's not recommended to update to Malcolm v23.05.0 from a previous version of Malcolm. Instead, shut down Malcolm, rename your old Malcolm installation directory to something else, and reconfigure Malcolm using `./scripts/configure` and `./scripts/auth_setup`.

v23.04.0...v23.05.0

* New features
    - integrate [ICSNPP-Synchrophasor](https://github.com/cisagov/icsnpp-synchrophasor/) parser (idaholab#190)
    - [End-to-end Malcolm and Hedgehog Linux ISO Installation((cisagov://cisagov.github.io/Malcolm/docs/malcolm-hedgehog-e2e-iso-install.html#InstallationExample) document (idaholab#181)
    - support Malcolm deployment with Kubernetes (idaholab#149)
        + see [Deploying Malcolm with Kubernetes](https://idaholab.github.io/Malcolm/docs/kubernetes.html#Kubernetes)
        + This could be considered a "beta" release for Malcolm deployment with Kubernetes, as there is still [some work](https://github.com/idaholab/Malcolm/issues?q=is%3Aissue+is%3Aopen+kubernetes) to be done in this area. Please let us know what issues or suggestions you have via the [issue tracker](https://github.com/idaholab/Malcolm/issues) or via email to [malcolm@inl.gov](mailto:malcolm@inl.gov?subject=Malcolm).
        + contributing issues:
            * inotify issue (idaholab#168)
            * htadmin/nginx and htpasswd (idaholab#169)
            * opensearch (idaholab#170)
            * uploading large PCAP files (idaholab#171)
            * script consolidation (idaholab#172)
            * documentation (idaholab#173)
            * user-defined persistent volumes (idaholab#174)
            * opensearch keystore (idaholab#176)
            * expose other TCP services (idaholab#183)
            * provide with filebeat access to nginx access and error logs (idaholab#186)
            * use Secrets for some environment variables instead of ConfigMaps (idaholab#189)

* Enhancements and fixes
    * remove name-map-ui container (idaholab#165) in favor of using NetBox for asset identification
    * Python script refactoring, consolidation and cleanup
    * standardization of Docker container entrypoints
    * create `./scripts/configure` alias for `./scripts/install.py --configure`

* Component version updates
    - Arkime to [v4.3.0](https://github.com/arkime/arkime/blob/fa0db2415bdc109be7a4dd8ee2c2838673980b5f/CHANGELOG#L33-L72)
    - Capa to [v5.1.0](https://github.com/mandiant/capa/releases/tag/v5.1.0)
    - Fluent Bit to [v2.1.2](https://fluentbit.io/announcements/v2.1.2/)
    - NetBox to [v3.5.0](https://github.com/netbox-community/netbox/releases/tag/v3.5.0)
    - NGINX to [v1.22.1](http://nginx.org/en/CHANGES-1.22)
    - Supercronic to [v0.2.24](https://github.com/aptible/supercronic/releases/tag/v0.2.24)
    - Suricata to [v6.0.10](https://suricata.io/2023/01/31/suricata-6-0-10-released/)
    - Yara to [v4.3.0](https://github.com/VirusTotal/yara/releases/tag/v4.3.0)
    - Zeek to [v5.2.1](https://github.com/zeek/zeek/releases/tag/v5.2.1)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@mmguero mmguero

Labels
cloud Relating to deployment of Malcolm in the cloud and/or with Kubernetes install.py Relating to the install.py configuration script
Projects
Malcolm
Status: Released
Milestone
v23.05.0
Development

No branches or pull requests
1 participant
@mmguero