Merge remote-tracking branch 'origin/dev' into fix/env-group-conundrum

.gitignore

@@ -3,3 +3,4 @@ dist
 .idea
 .run
 /myks
+**/.myks/tmp

CHANGELOG.md

@@ -1,3 +1,27 @@
+# [2.1.0](https://github.com/mykso/myks/compare/v2.0.4...v2.1.0) (2023-10-31)
+
+
+### Bug Fixes
+
+* **init:** ignore top level .myks folder ([#116](https://github.com/mykso/myks/issues/116)) ([b39eb8d](https://github.com/mykso/myks/commit/b39eb8df0a9cc22524314431b61bb7474608d270))
+* **render:** fix panic if metadata.name is not set ([#123](https://github.com/mykso/myks/issues/123)) ([cdf3266](https://github.com/mykso/myks/commit/cdf3266132064b5f3627714199227cd75ed4cfa9))
+* **smart-mode:** correct detection of changes in "_env/argocd" ([#111](https://github.com/mykso/myks/issues/111)) ([ec5dd1d](https://github.com/mykso/myks/commit/ec5dd1d99b7fddf2a452c8b86609775e8222b851))
+* **smart-mode:** filter out deleted envs and apps ([93fa2e3](https://github.com/mykso/myks/commit/93fa2e3786250b4e0c305af0923204d2bf1e9341)), closes [#114](https://github.com/mykso/myks/issues/114)
+
+
+### Features
+
+* add metadata to ytt steps ([#119](https://github.com/mykso/myks/issues/119)) ([6443a20](https://github.com/mykso/myks/commit/6443a2097dc061074a9c6f3ca019a96054c51f4f))
+* **global-ytt:** log ytt output on error ([#120](https://github.com/mykso/myks/issues/120)) ([a22f7aa](https://github.com/mykso/myks/commit/a22f7aa0ea9dc27985e906c6f0f5213cddcd6697))
+* **ui:** add smart-mode.only-print flag for debugging ([0a6349a](https://github.com/mykso/myks/commit/0a6349a974fca2317f7d51adae337a3dce0dc4c0))
+
+## [2.0.4](https://github.com/mykso/myks/compare/v2.0.3...v2.0.4) (2023-09-29)
+
+
+### Bug Fixes
+
+* **release:** trigger ([b6dd8bb](https://github.com/mykso/myks/commit/b6dd8bba5458b004c6e58c3365a6e969f2222db2))
+
 ## [2.0.3](https://github.com/mykso/myks/compare/v2.0.2...v2.0.3) (2023-09-29)
 
 

README.md

@@ -2,7 +2,49 @@
 
 **M**anage (**my**) **y**aml for **K**ubernetes **s**imply. Or something like that.
 
-**Myks** helps to maintain configuration of many applications for multiple Kubernetes clusters.
+**Myks** is a tool and a framework for managing configuration of applications for multiple Kubernetes clusters.
+
+## Why?
+
+- We needed to manage configuration of multiple applications for multiple clusters.
+- The clusters had different flavors (e.g. k3s, Redshift, AKS) and different versions of Kubernetes.
+- The applications were distributed in various forms and rendered with different tools (e.g. Helm, kustomize, plain YAML).
+- We wanted to be able to automatically update third-party components (e.g. kube-prometheus-stack, ArgoCD)
+  and to be able to pin versions if needed.
+- And the most important thing: we wanted to have final rendered manifests in git. So that we can review them, audit them,
+  and have a single source of truth.
+
+## How does it work?
+
+Myks consumes a set of templates and values and renders them into a set of Kubernetes manifests.
+It heavily relies on [ytt](https://get-ytt.io/) and [vendir](https://carvel.dev/vendir/) under the hood.
+
+Here's a quick example:
+
+```console
+$ # Switch to an empty directory
+$ cd $(mktemp -d)
+$ # Initialize a new project with example configuration
+$ myks init
+$ # Optionally, check the generated files
+$ find
+$ # Sync and render everything
+$ myks all
+$ # Check the rendered manifests
+$ find rendered
+```
+
+## Usage
+
+### Running `sync` against protected repositories and registries
+
+Vendir uses `secret` resources to authenticate against protected repositories.
+These are references by the `vendir.yaml` with the `secretRef` key.
+
+Myks dynamically creates these secrets based on environment variables prefixed with `VENDIR_SECRET_`.
+For example, if you reference a secret named "mycreds" in your `vendir.yaml`,
+you need to define the environment variables `VENDIR_SECRET_MYCREDS_USERNAME` and `VENDIR_SECRET_MYCREDS_PASSWORD`.
+The secrets are cleaned up automatically after the sync is complete.
 
 ## Development
 
@@ -45,12 +87,3 @@ $ find
 $ # Sync and render everything
 $ myks all envs --log-level debug
 ```
-
-### Run
-
-#### Running `sync` against protected repositories and registries
-
-Vendir uses `secret` resources to authenticate against protected repositories. These are references by the `vendir.yaml` with the `secretRef` key. 
-
-Myks dynamically creates these secrets based on environment variables prefixed with `VENDIR_SECRET_`.
-For example, if you reference a secret named "mycreds" in your `vendir.yaml, you need to define the environment variables VENDIR_SECRET_MYCREDS_USERNAME` and `VENDIR_SECRET_MYCREDS_PASSWORD`. The secrets are cleaned up automatically after the sync is complete.

cmd/all.go

@@ -16,28 +16,32 @@ func init() {
 			ANNOTATION_SMART_MODE: ANNOTATION_TRUE,
 		},
 		Run: func(cmd *cobra.Command, args []string) {
-			g := myks.New(".")
-
-			if err := g.ValidateRootDir(); err != nil {
-				log.Fatal().Err(err).Msg("Root directory is not suitable for myks")
-			}
-
-			if err := g.Init(asyncLevel, envAppMap); err != nil {
-				log.Fatal().Err(err).Msg("Unable to initialize myks's globe")
-			}
-
-			if err := g.SyncAndRender(asyncLevel); err != nil {
-				log.Fatal().Err(err).Msg("Unable to sync vendir configs")
-			}
-
-			// Cleaning up only if all environments and applications were processed
-			if envAppMap == nil {
-				if err := g.Cleanup(); err != nil {
-					log.Fatal().Err(err).Msg("Unable to cleanup")
-				}
-			}
+			RunAllCmd()
 		},
 	}
 
 	rootCmd.AddCommand(cmd)
 }
+
+func RunAllCmd() {
+	g := myks.New(".")
+
+	if err := g.ValidateRootDir(); err != nil {
+		log.Fatal().Err(err).Msg("Root directory is not suitable for myks")
+	}
+
+	if err := g.Init(asyncLevel, envAppMap); err != nil {
+		log.Fatal().Err(err).Msg("Unable to initialize myks's globe")
+	}
+
+	if err := g.SyncAndRender(asyncLevel); err != nil {
+		log.Fatal().Err(err).Msg("Unable to sync vendir configs")
+	}
+
+	// Cleaning up only if all environments and applications were processed
+	if envAppMap == nil {
+		if err := g.Cleanup(); err != nil {
+			log.Fatal().Err(err).Msg("Unable to cleanup")
+		}
+	}
+}

cmd/root.go

@@ -8,6 +8,7 @@ import (
 	"strings"
 	"time"
 
+	aurora "github.com/logrusorgru/aurora/v4"
 	"github.com/rs/zerolog"
 	"github.com/rs/zerolog/log"
 	"github.com/spf13/cobra"
@@ -57,6 +58,9 @@ func init() {
 		"if not provided, only local changes will be considered"
 	rootCmd.PersistentFlags().String("smart-mode.base-revision", "", smartModeBaseRevisionHelp)
 
+	smartModeOnlyPrintHelp := "only print the list of environments and applications that would be rendered in Smart Mode"
+	rootCmd.PersistentFlags().Bool("smart-mode.only-print", false, smartModeOnlyPrintHelp)
+
 	if err := viper.BindPFlags(rootCmd.PersistentFlags()); err != nil {
 		log.Fatal().Err(err).Msg("Unable to bind flags")
 	}
@@ -125,6 +129,21 @@ func detectTargetEnvsAndApps(cmd *cobra.Command, args []string) (err error) {
 		Interface("envAppMap", envAppMap).
 		Msg("Parsed arguments")
 
+	if viper.GetBool("smart-mode.only-print") {
+		fmt.Println(aurora.Bold("\nSmart Mode detected:"))
+		for env, apps := range envAppMap {
+			fmt.Printf("→ %s\n", env)
+			if apps == nil {
+				fmt.Println(aurora.Bold("    ALL"))
+			} else {
+				for _, app := range apps {
+					fmt.Printf("    %s\n", app)
+				}
+			}
+		}
+		os.Exit(0)
+	}
+
 	return nil
 }
 

examples/default/.myks.yaml

@@ -0,0 +1,7 @@
+---
+# Sets the number of applications to be processed in parallel.
+# The default (0) is no limit.
+async: 0
+# One of the zerolog log levels.
+# See: https://github.com/rs/zerolog#leveled-logging
+log-level: info

examples/default/envs/.gitignore

@@ -0,0 +1 @@
+**/.myks/tmp/

examples/default/envs/_env/argocd/annotations.overlay.ytt.yaml

@@ -0,0 +1,12 @@
+#@ load("@ytt:overlay", "overlay")
+#@ load("@ytt:data", "data")
+
+#! Match all ArgoCD resources: Application, AppProject, Secret.
+#@overlay/match by=overlay.all, expects="1+"
+---
+#@overlay/match missing_ok=True
+#@overlay/match-child-defaults missing_ok=True
+metadata:
+  annotations:
+    myks.dev/environment: #@ data.values.environment.id
+    app.kubernetes.io/source: #@ data.values.myks.gitRepoUrl

examples/default/envs/_env/argocd/secret.overlay.ytt.yaml

@@ -0,0 +1,15 @@
+#@ load("@ytt:overlay", "overlay")
+---
+#@ def secret_fragment():
+kind: Secret
+metadata:
+  labels:
+    argocd.argoproj.io/secret-type: cluster
+#@ end
+
+#@overlay/match by=overlay.subset(secret_fragment()), expects="0+"
+---
+#! See https://argo-cd.readthedocs.io/en/release-2.8/operator-manual/declarative-setup/#clusters
+stringData:
+  config: ARGOCD_CLUSTER_CONNECT_CONFIG
+  server: ARGOCD_CLUSTER_SERVER_URL

examples/default/envs/_env/ytt/common.ytt.yaml

@@ -0,0 +1,17 @@
+#! This file contains YTT overlays for common modifications to Kubernetes resources.
+
+#@ load("@ytt:data", "data")
+#@ load("@ytt:overlay", "overlay")
+
+#! Add common labels and annotations to all resources.
+#! ------------------------------------------------------------
+#@overlay/match by=lambda i, l, r: "metadata" in l, when="1+"
+---
+#@overlay/match-child-defaults missing_ok=True
+metadata:
+  #! See https://ambassadorlabs.github.io/k8s-for-humans/
+  annotations:
+    a8r.io/repository: #@ data.values.myks.gitRepoUrl
+    #! More examples:
+    #! a8r.io/owner: https://github.com/mykso
+    #! a8r.io/chat: #@ data.values.application.chat

examples/default/envs/env-data.ytt.yaml

@@ -0,0 +1,28 @@
+#! This is an example of a data values file for the root environment.
+#! All nested environments inherit these values. The values can be overridden in nested environments.
+#! This file is a good place to define default values for all environments.
+#!
+#! To change the schema of the data values, use `data/values-schema` annotation instead of `data/values`.
+#! Refer to the documentation of ytt overlays and data values for more information.
+
+#@data/values
+---
+argocd:
+  namespace: system-argocd
+  app:
+    prefix: app-
+    #! Disable finalizers to preserve resources after deleting the ArgoCD application.
+    finalizers: []
+    source:
+      plugin:
+        name: argocd-vault-plugin-v1.0.0
+      #! Fixed config to run tests successfull in pipeline
+      targetRevision: main
+      repoURL: git@github.com:mykso/myks.git
+  project:
+    prefix: env-
+
+#! Fixed git config to run tests successfull in pipeline.
+myks:
+  gitRepoBranch: "main"
+  gitRepoUrl: "git@github.com:mykso/myks.git"

examples/default/envs/mykso/dev/_apps/argocd/.myks/vendir.lock.yaml

@@ -0,0 +1,11 @@
+apiVersion: vendir.k14s.io/v1alpha1
+directories:
+- contents:
+  - git:
+      commitTitle: Bump version to 2.7.3 (#13719)...
+      sha: e7891b899a35dca06ae94965ea5ae2a86b344848
+      tags:
+      - v2.7.3
+    path: .
+  path: ytt/argocd
+kind: LockConfig

examples/default/envs/mykso/dev/_apps/argocd/.myks/vendir.sync.yaml

@@ -0,0 +1 @@
+ytt/argocd: c9846652945082356eb1bde564cd69e4c2162b49b698ffc04a5225d93d532f69

examples/default/envs/mykso/dev/_apps/argocd/.myks/vendir.yaml

@@ -0,0 +1,12 @@
+apiVersion: vendir.k14s.io/v1alpha1
+kind: Config
+directories:
+- path: ytt/argocd
+  contents:
+  - path: .
+    git:
+      url: https://github.com/argoproj/argo-cd
+      ref: v2.7.3
+    includePaths:
+    - manifests/ha/install.yaml
+    newRootPath: manifests/ha

examples/default/envs/mykso/dev/_apps/argocd/app-data.ytt.yaml

@@ -0,0 +1,5 @@
+#@data/values-schema
+---
+#@overlay/match-child-defaults missing_ok=True
+application:
+  gcpServiceAccountEmail: argocd-repo-server@mykso-dev.iam.gserviceaccount.com