From 45f4fd63e3ae57e97ad9d6b5c6c1dcaf14713730 Mon Sep 17 00:00:00 2001 From: Jim Grady Date: Tue, 15 Nov 2022 14:31:45 -0500 Subject: [PATCH 01/12] wip: CI/CD documentation --- README.md | 35 +++++++++++++++++++++++++++++++++-- 1 file changed, 33 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 49025bdb55..cc9b19f84b 100644 --- a/README.md +++ b/README.md @@ -67,8 +67,10 @@ A rapid word collection tool. See the [User Guide](https://sillsdev.github.io/Th 1. [Development Environment](#development-environment) 2. [Kubernetes Environment](#kubernetes-environment) 6. [User Guide](#user-guide) -7. [Production](#production) -8. [Learn More](#learn-more) +7. [Continuous Integration](#continuous-integration) +8. [Continuous Deployment](#continuous-deployment) +9. [Production](#production) +10. [Learn More](#learn-more) ## Getting Started with Development @@ -953,6 +955,35 @@ To locally build the user guide statically into `docs/user-guide/site`: tox -e user-guide ``` +## Continuous Integration + +The Continuous Integration (CI) processes are initiated when a Pull Request (PR) is created. For each push to the PR +branch, a set of CI tests are run. When all the CI tests pass _and_ the PR changes have been reviewed and approved by a +team member, then the PR may be merged into the `master` branch. When the merge is complete, _The Combine_ software is +built and deployed to the QA server: + +```mermaid +sequenceDiagram + participant: D1 as Developer 1 + participant: D2 Developer 2 + participant: PR + participant: master as Master branch + participant: GH as GitHub Runner + participant: SH as Self-Hosted Runner + D1 ->> PR: create + activate PR + PR ->> D2: request review + PR ->> GH: start CI tests + activate GH + +``` + +## Continuous Deployment + +_The Combine_ uses _GitHub Actions_ to implement its Continuous Integration and Continuous Deployment processes. New +features and bug fixes are implemented on a developer's private branch. When the changes have been implemented, a Pull +Request is created which triggers the Continuous Integration process. + ## Production The process for configuring and deploying _TheCombine_ for production targets is described in From fa4e132f18398b77b999120027699dcd5e9af544 Mon Sep 17 00:00:00 2001 From: Jim Grady Date: Tue, 15 Nov 2022 14:52:35 -0500 Subject: [PATCH 02/12] Update to CI process --- README.md | 31 +++++++++++++++++++++++-------- 1 file changed, 23 insertions(+), 8 deletions(-) diff --git a/README.md b/README.md index cc9b19f84b..8d11c80d59 100644 --- a/README.md +++ b/README.md @@ -964,17 +964,32 @@ built and deployed to the QA server: ```mermaid sequenceDiagram - participant: D1 as Developer 1 - participant: D2 Developer 2 - participant: PR - participant: master as Master branch - participant: GH as GitHub Runner - participant: SH as Self-Hosted Runner + participant D1 as Developer 1 + participant D2 as Developer 2 + participant PR + participant master as Master branch + participant GH as GitHub Runner + participant SH as Self-Hosted Runner + participant reg as AWS Private Registry D1 ->> PR: create activate PR - PR ->> D2: request review - PR ->> GH: start CI tests activate GH + par + PR ->> D2: request review + D2 ->> PR: Approved + and + PR ->> GH: start CI tests + GH ->> PR: all tests pass + end + deactivate GH + PR ->> master: merge changes + activate master + PR ->> PR: delete branch + deactivate PR + master ->> GH: build The Combine + master ->> reg: Push images + master ->> SH: Deploy to QA server + deactivate master ``` From a3dc639f144ca8b0881324f95e1128a2f9af3bdf Mon Sep 17 00:00:00 2001 From: Jim Grady Date: Wed, 16 Nov 2022 08:33:24 -0500 Subject: [PATCH 03/12] Complete CI/CD diagrams --- README.md | 53 +++++++++++++++++++++++++++++++++++------------------ 1 file changed, 35 insertions(+), 18 deletions(-) diff --git a/README.md b/README.md index 8d11c80d59..1d9f9a4eef 100644 --- a/README.md +++ b/README.md @@ -67,7 +67,7 @@ A rapid word collection tool. See the [User Guide](https://sillsdev.github.io/Th 1. [Development Environment](#development-environment) 2. [Kubernetes Environment](#kubernetes-environment) 6. [User Guide](#user-guide) -7. [Continuous Integration](#continuous-integration) +7. [Continuous Integration and Continuous Deployment](#continuous-integration-and-continuous-deployment) 8. [Continuous Deployment](#continuous-deployment) 9. [Production](#production) 10. [Learn More](#learn-more) @@ -955,31 +955,33 @@ To locally build the user guide statically into `docs/user-guide/site`: tox -e user-guide ``` -## Continuous Integration +## Continuous Integration and Continuous Deployment -The Continuous Integration (CI) processes are initiated when a Pull Request (PR) is created. For each push to the PR -branch, a set of CI tests are run. When all the CI tests pass _and_ the PR changes have been reviewed and approved by a -team member, then the PR may be merged into the `master` branch. When the merge is complete, _The Combine_ software is -built and deployed to the QA server: +### On Pull Request + +When a Pull Request (PR) is created and for each push to the PR branch, a set of CI tests are run. When all the CI tests +pass _and_ the PR changes have been reviewed and approved by a team member, then the PR may be merged into the `master` +branch. When the merge is complete, _The Combine_ software is built, pushed to the AWS ECR Private registry, and +deployed to the QA server: ```mermaid sequenceDiagram - participant D1 as Developer 1 - participant D2 as Developer 2 + actor Author + actor Reviewer participant PR participant master as Master branch participant GH as GitHub Runner participant SH as Self-Hosted Runner participant reg as AWS Private Registry - D1 ->> PR: create + Author ->> PR: create activate PR activate GH par - PR ->> D2: request review - D2 ->> PR: Approved - and - PR ->> GH: start CI tests + PR ->> GH: run CI tests GH ->> PR: all tests pass + and + PR ->> Reviewer: request review + Reviewer ->> PR: Approved end deactivate GH PR ->> master: merge changes @@ -990,14 +992,29 @@ sequenceDiagram master ->> reg: Push images master ->> SH: Deploy to QA server deactivate master - ``` -## Continuous Deployment +### On Release -_The Combine_ uses _GitHub Actions_ to implement its Continuous Integration and Continuous Deployment processes. New -features and bug fixes are implemented on a developer's private branch. When the changes have been implemented, a Pull -Request is created which triggers the Continuous Integration process. +When a team member creates a release on _The Combine's_ GitHub project page, a Release tag is created on the master +branch, the software is built and pushed to the AWS ECR Public registry and then deployed to the production server. + +```mermaid +sequenceDiagram + actor Developer + participant Release + participant master as Master branch + participant GH as GitHub Runner + participant SH as Self-Hosted Runner + participant reg as AWS Public Registry + Developer ->> Release: create + Release ->> master: Create release tag + activate master + master ->> GH: build The Combine + master ->> reg: Push images + master ->> SH: Deploy to Production server + deactivate master +``` ## Production From 5db8d8e08b425f45fc72daa41ce26f3364570d68 Mon Sep 17 00:00:00 2001 From: Jim Grady Date: Wed, 16 Nov 2022 08:36:24 -0500 Subject: [PATCH 04/12] Update TOC --- README.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index 1d9f9a4eef..72de698646 100644 --- a/README.md +++ b/README.md @@ -68,9 +68,10 @@ A rapid word collection tool. See the [User Guide](https://sillsdev.github.io/Th 2. [Kubernetes Environment](#kubernetes-environment) 6. [User Guide](#user-guide) 7. [Continuous Integration and Continuous Deployment](#continuous-integration-and-continuous-deployment) -8. [Continuous Deployment](#continuous-deployment) -9. [Production](#production) -10. [Learn More](#learn-more) + 1. [On Pull Request](#on-pull-request) + 2. [On Release](#on-release) +8. [Production](#production) +9. [Learn More](#learn-more) ## Getting Started with Development From d38f65da1717bcb46dd2ccfe1d108e82b77878f9 Mon Sep 17 00:00:00 2001 From: Jim Grady Date: Wed, 16 Nov 2022 10:55:09 -0500 Subject: [PATCH 05/12] Update sequence diagrams to show that images are pushed from GH runner --- README.md | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 72de698646..1b24134783 100644 --- a/README.md +++ b/README.md @@ -990,7 +990,10 @@ sequenceDiagram PR ->> PR: delete branch deactivate PR master ->> GH: build The Combine - master ->> reg: Push images + activate GH + GH ->> reg: Push images + GH ->> master: build complete + deactivate GH master ->> SH: Deploy to QA server deactivate master ``` @@ -1012,7 +1015,10 @@ sequenceDiagram Release ->> master: Create release tag activate master master ->> GH: build The Combine - master ->> reg: Push images + activate GH + GH ->> reg: Push images + GH ->> master: build complete + deactivate GH master ->> SH: Deploy to Production server deactivate master ``` From f3cc2f97b5f294e1f5f522f3606aee345846d5dc Mon Sep 17 00:00:00 2001 From: Jim Grady Date: Wed, 16 Nov 2022 20:36:10 -0500 Subject: [PATCH 06/12] Update CI/CD sequence diagrams --- README.md | 89 +++++++++++++++++++++++++++++++++---------------------- 1 file changed, 53 insertions(+), 36 deletions(-) diff --git a/README.md b/README.md index 1b24134783..1383fe0aa0 100644 --- a/README.md +++ b/README.md @@ -969,33 +969,43 @@ deployed to the QA server: sequenceDiagram actor Author actor Reviewer - participant PR - participant master as Master branch - participant GH as GitHub Runner - participant SH as Self-Hosted Runner + participant github as sillsdev/TheCombine + participant gh_runner as GitHub Runner + participant sh_runner as Self-Hosted Runner participant reg as AWS Private Registry - Author ->> PR: create - activate PR - activate GH + participant server as QA Server + Author ->> github: create Pull Request(work_branch) + activate github par - PR ->> GH: run CI tests - GH ->> PR: all tests pass + loop for each CI test + github ->> gh_runner: start CI test + activate gh_runner + gh_runner ->> gh_runner: checkout work_branch + gh_runner ->> gh_runner: run test + gh_runner -->> github: test passed + deactivate gh_runner + end and - PR ->> Reviewer: request review - Reviewer ->> PR: Approved + github ->> Reviewer: request review + Reviewer -->> github: Approved end - deactivate GH - PR ->> master: merge changes - activate master - PR ->> PR: delete branch - deactivate PR - master ->> GH: build The Combine - activate GH - GH ->> reg: Push images - GH ->> master: build complete - deactivate GH - master ->> SH: Deploy to QA server - deactivate master + github ->> github: merge work_branch to master + github ->> github: delete work_branch + github ->> gh_runner: run deploy_qa workflow + activate gh_runner + gh_runner ->> gh_runner: checkout master + gh_runner ->> gh_runner: build The Combine + gh_runner ->> reg: Push images + gh_runner ->> github: build complete(image_tag) + deactivate gh_runner + github ->> sh_runner: Deploy to QA server (image_tag) + activate sh_runner + loop frontend, backend, database, maintenance + sh_runner ->> server: update deployment image(image_tag) + server ->> reg: pull image(image_tag) + reg ->> server: updated image(image_tag) + end + deactivate sh_runner ``` ### On Release @@ -1007,20 +1017,27 @@ branch, the software is built and pushed to the AWS ECR Public registry and then sequenceDiagram actor Developer participant Release - participant master as Master branch - participant GH as GitHub Runner - participant SH as Self-Hosted Runner + participant gh_runner as GitHub Runner + participant sh_runner as Self-Hosted Runner participant reg as AWS Public Registry - Developer ->> Release: create - Release ->> master: Create release tag - activate master - master ->> GH: build The Combine - activate GH - GH ->> reg: Push images - GH ->> master: build complete - deactivate GH - master ->> SH: Deploy to Production server - deactivate master + participant server as Production Server + Developer ->> github: create Release + github ->> github: Create release tag on master branch + github ->> gh_runner: run deploy_release workflow + activate gh_runner + gh_runner ->> gh_runner: checkout release tag + gh_runner ->> gh_runner: build The Combine + gh_runner ->> reg: Push images + gh_runner ->> github: build complete(image_tag) + deactivate gh_runner + github ->> sh_runner: Deploy to Production server (image_tag) + activate sh_runner + loop frontend, backend, database, maintenance + sh_runner ->> server: update deployment image(image_tag) + server ->> reg: pull image(image_tag) + reg ->> server: updated image(image_tag) + end + deactivate sh_runner ``` ## Production From fcefee9127651fc3057ccf07bf07ec7b8e2497e1 Mon Sep 17 00:00:00 2001 From: Jim Grady Date: Thu, 17 Nov 2022 05:58:46 -0500 Subject: [PATCH 07/12] Update CI/CD sequence diagrams --- README.md | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/README.md b/README.md index 1383fe0aa0..58b45dd9c1 100644 --- a/README.md +++ b/README.md @@ -978,7 +978,8 @@ sequenceDiagram activate github par loop for each CI test - github ->> gh_runner: start CI test + Note over github,gh_runner: CI tests are run concurrently + github ->> gh_runner: start CI test activate gh_runner gh_runner ->> gh_runner: checkout work_branch gh_runner ->> gh_runner: run test @@ -996,14 +997,14 @@ sequenceDiagram gh_runner ->> gh_runner: checkout master gh_runner ->> gh_runner: build The Combine gh_runner ->> reg: Push images - gh_runner ->> github: build complete(image_tag) + gh_runner -->> github: build complete(image_tag) deactivate gh_runner github ->> sh_runner: Deploy to QA server (image_tag) activate sh_runner loop frontend, backend, database, maintenance sh_runner ->> server: update deployment image(image_tag) server ->> reg: pull image(image_tag) - reg ->> server: updated image(image_tag) + reg -->> server: updated image(image_tag) end deactivate sh_runner ``` @@ -1016,7 +1017,7 @@ branch, the software is built and pushed to the AWS ECR Public registry and then ```mermaid sequenceDiagram actor Developer - participant Release + participant githug as sillsdev/TheCombine participant gh_runner as GitHub Runner participant sh_runner as Self-Hosted Runner participant reg as AWS Public Registry @@ -1028,14 +1029,14 @@ sequenceDiagram gh_runner ->> gh_runner: checkout release tag gh_runner ->> gh_runner: build The Combine gh_runner ->> reg: Push images - gh_runner ->> github: build complete(image_tag) + gh_runner -->> github: build complete(image_tag) deactivate gh_runner github ->> sh_runner: Deploy to Production server (image_tag) activate sh_runner loop frontend, backend, database, maintenance sh_runner ->> server: update deployment image(image_tag) server ->> reg: pull image(image_tag) - reg ->> server: updated image(image_tag) + reg -->> server: updated image(image_tag) end deactivate sh_runner ``` From b4b5b92a598731afdae1bb44666154323e7b508e Mon Sep 17 00:00:00 2001 From: Jim Grady Date: Thu, 17 Nov 2022 06:00:49 -0500 Subject: [PATCH 08/12] Update CI/CD sequence diagrams --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 58b45dd9c1..70773c43bc 100644 --- a/README.md +++ b/README.md @@ -996,7 +996,7 @@ sequenceDiagram activate gh_runner gh_runner ->> gh_runner: checkout master gh_runner ->> gh_runner: build The Combine - gh_runner ->> reg: Push images + gh_runner ->> reg: Push images(image_tag) gh_runner -->> github: build complete(image_tag) deactivate gh_runner github ->> sh_runner: Deploy to QA server (image_tag) @@ -1028,7 +1028,7 @@ sequenceDiagram activate gh_runner gh_runner ->> gh_runner: checkout release tag gh_runner ->> gh_runner: build The Combine - gh_runner ->> reg: Push images + gh_runner ->> reg: Push images(image_tag) gh_runner -->> github: build complete(image_tag) deactivate gh_runner github ->> sh_runner: Deploy to Production server (image_tag) From 67cd560e8fce8d56fc74341125d5241a38654e20 Mon Sep 17 00:00:00 2001 From: Jim Grady Date: Thu, 17 Nov 2022 06:10:30 -0500 Subject: [PATCH 09/12] Clarify loop variable in CI/CD diagrams --- README.md | 28 +++++++++++++++++----------- 1 file changed, 17 insertions(+), 11 deletions(-) diff --git a/README.md b/README.md index 70773c43bc..50810b4d50 100644 --- a/README.md +++ b/README.md @@ -994,14 +994,17 @@ sequenceDiagram github ->> github: delete work_branch github ->> gh_runner: run deploy_qa workflow activate gh_runner - gh_runner ->> gh_runner: checkout master - gh_runner ->> gh_runner: build The Combine - gh_runner ->> reg: Push images(image_tag) - gh_runner -->> github: build complete(image_tag) + loop component in (frontend, backend, database, maintenance) + Note right of gh_runner: components are built concurrently + gh_runner ->> gh_runner: checkout master + gh_runner ->> gh_runner: build component + gh_runner ->> reg: Push component image(image_tag) + gh_runner -->> github: build complete(image_tag) + end deactivate gh_runner github ->> sh_runner: Deploy to QA server (image_tag) activate sh_runner - loop frontend, backend, database, maintenance + loop deployment in (frontend, backend, database, maintenance) sh_runner ->> server: update deployment image(image_tag) server ->> reg: pull image(image_tag) reg -->> server: updated image(image_tag) @@ -1017,7 +1020,7 @@ branch, the software is built and pushed to the AWS ECR Public registry and then ```mermaid sequenceDiagram actor Developer - participant githug as sillsdev/TheCombine + participant github as sillsdev/TheCombine participant gh_runner as GitHub Runner participant sh_runner as Self-Hosted Runner participant reg as AWS Public Registry @@ -1026,14 +1029,17 @@ sequenceDiagram github ->> github: Create release tag on master branch github ->> gh_runner: run deploy_release workflow activate gh_runner - gh_runner ->> gh_runner: checkout release tag - gh_runner ->> gh_runner: build The Combine - gh_runner ->> reg: Push images(image_tag) - gh_runner -->> github: build complete(image_tag) + loop component in (frontend, backend, database, maintenance) + Note right of gh_runner: components are built concurrently + gh_runner ->> gh_runner: checkout release tag + gh_runner ->> gh_runner: build component + gh_runner ->> reg: Push component image(image_tag) + gh_runner -->> github: build complete(image_tag) + end deactivate gh_runner github ->> sh_runner: Deploy to Production server (image_tag) activate sh_runner - loop frontend, backend, database, maintenance + loop deployment in (frontend, backend, database, maintenance) sh_runner ->> server: update deployment image(image_tag) server ->> reg: pull image(image_tag) reg -->> server: updated image(image_tag) From 885f228084b514d4017eba6215651558aa179f7e Mon Sep 17 00:00:00 2001 From: Jim Grady Date: Thu, 17 Nov 2022 08:32:36 -0500 Subject: [PATCH 10/12] Make message case consistent in CI/CD diagrams --- README.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/README.md b/README.md index 50810b4d50..07016d200b 100644 --- a/README.md +++ b/README.md @@ -988,7 +988,7 @@ sequenceDiagram end and github ->> Reviewer: request review - Reviewer -->> github: Approved + Reviewer -->> github: approved end github ->> github: merge work_branch to master github ->> github: delete work_branch @@ -998,11 +998,11 @@ sequenceDiagram Note right of gh_runner: components are built concurrently gh_runner ->> gh_runner: checkout master gh_runner ->> gh_runner: build component - gh_runner ->> reg: Push component image(image_tag) + gh_runner ->> reg: push component image(image_tag) gh_runner -->> github: build complete(image_tag) end deactivate gh_runner - github ->> sh_runner: Deploy to QA server (image_tag) + github ->> sh_runner: deploy to QA server (image_tag) activate sh_runner loop deployment in (frontend, backend, database, maintenance) sh_runner ->> server: update deployment image(image_tag) @@ -1026,18 +1026,18 @@ sequenceDiagram participant reg as AWS Public Registry participant server as Production Server Developer ->> github: create Release - github ->> github: Create release tag on master branch + github ->> github: create release tag on master branch github ->> gh_runner: run deploy_release workflow activate gh_runner loop component in (frontend, backend, database, maintenance) Note right of gh_runner: components are built concurrently gh_runner ->> gh_runner: checkout release tag gh_runner ->> gh_runner: build component - gh_runner ->> reg: Push component image(image_tag) + gh_runner ->> reg: push component image(image_tag) gh_runner -->> github: build complete(image_tag) end deactivate gh_runner - github ->> sh_runner: Deploy to Production server (image_tag) + github ->> sh_runner: deploy to Production server (image_tag) activate sh_runner loop deployment in (frontend, backend, database, maintenance) sh_runner ->> server: update deployment image(image_tag) From c60126b1f9b642994dc9a5740b0856fdd4e2e89d Mon Sep 17 00:00:00 2001 From: Jim Grady Date: Thu, 17 Nov 2022 08:36:35 -0500 Subject: [PATCH 11/12] make deployment updates asynchronous calls --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 07016d200b..ca459fb23a 100644 --- a/README.md +++ b/README.md @@ -1005,7 +1005,7 @@ sequenceDiagram github ->> sh_runner: deploy to QA server (image_tag) activate sh_runner loop deployment in (frontend, backend, database, maintenance) - sh_runner ->> server: update deployment image(image_tag) + sh_runner -) server: update deployment image(image_tag) server ->> reg: pull image(image_tag) reg -->> server: updated image(image_tag) end @@ -1040,7 +1040,7 @@ sequenceDiagram github ->> sh_runner: deploy to Production server (image_tag) activate sh_runner loop deployment in (frontend, backend, database, maintenance) - sh_runner ->> server: update deployment image(image_tag) + sh_runner -) server: update deployment image(image_tag) server ->> reg: pull image(image_tag) reg -->> server: updated image(image_tag) end From 86ff9ca2fe8fdeab22142b0eadb8e210f5aad04e Mon Sep 17 00:00:00 2001 From: Jim Grady Date: Fri, 18 Nov 2022 11:02:32 -0500 Subject: [PATCH 12/12] Remove superfluous spaces in ci/cd diagram --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index ca459fb23a..4678b2fb81 100644 --- a/README.md +++ b/README.md @@ -1002,7 +1002,7 @@ sequenceDiagram gh_runner -->> github: build complete(image_tag) end deactivate gh_runner - github ->> sh_runner: deploy to QA server (image_tag) + github ->> sh_runner: deploy to QA server(image_tag) activate sh_runner loop deployment in (frontend, backend, database, maintenance) sh_runner -) server: update deployment image(image_tag) @@ -1037,7 +1037,7 @@ sequenceDiagram gh_runner -->> github: build complete(image_tag) end deactivate gh_runner - github ->> sh_runner: deploy to Production server (image_tag) + github ->> sh_runner: deploy to Production server(image_tag) activate sh_runner loop deployment in (frontend, backend, database, maintenance) sh_runner -) server: update deployment image(image_tag)