Wait for applied objects to become healthy using CEL readiness rule or kstatus #2419

barney-s · 2024-08-05T18:43:57Z

Change description

apply + wait for ready
- Change Applier to get status using CEL rules if rule present
- fallback Logic to get status of applied objects using kstatus
- Applier.AreResourcesReady() wraps readiness checks
- Each stage is applied and then checks if resources are ready (not blocking reconciler)
- If resources are not ready, reconciler returns with an error causing
  a queued reconcile.
CEL based Readiness Rule support
- Change Composition spec to take a list of CEL rules for custom status
  rules
- pkg/cel/engine.go - Simple CEL wrapper to run rules against
Testcases:
- TestFirstCompositionTeamPage: applies teampage sample (will tests waiting for
  deployment becoming healthy)
- TestCustomStatusCELRule: tests using a CEL readiness rule for
  Deployment

Sample outputs

CEL rule based checks

CEL rule in TestCustomStatusCELRule/input.yaml: readyIf: "status.readyReplicas==status.replicas"

plan status showing CEL rule result:

status:
  compositionGeneration: 1
  compositionUID: 9e12a45d-8a38-4e4a-bc31-488934bdd211
  conditions:
  - lastTransitionTime: "2024-08-07T21:14:55Z"
    message: 'Evaluated and Applied stages: server'
    reason: ProcessedAllStages
    status: "True"
    type: Ready
  generation: 2
  inputGeneration: 1
  stages:
    server:
      appliedCount: 3
      lastApplied:
      - group: apps
        health: Healthy
        kind: Deployment
        name: team-landing
        namespace: my-team
        status: Readiness Rule evaluated to true
        version: v1
      - health: Healthy
        kind: Service
        name: team-landing-landing
        namespace: my-team
        status: Service is ready
        version: v1
      - health: Healthy
        kind: ConfigMap
        name: team-landing-page
        namespace: my-team
        status: Resource is always ready
        version: v1
      resourceCount: 3

kstatus based checks

plan status showing waiting for resources to become healthy:

status:
    compositionGeneration: 0
    conditions:
    - lastTransitionTime: "2024-08-06T17:06:53Z"
      message: 'Evaluated stages: server'
      reason: EvaluationPending
      status: "False"
      type: Ready
    - lastTransitionTime: "2024-08-06T17:06:53Z"
      message: 'Expander: server, Message: Not all resources are healthy'
      reason: WaitingForAppliedResources
      status: "True"
      type: Waiting
    inputGeneration: 0
    stages:
      server:
        appliedCount: 3
        lastApplied:
        - group: apps
          health: Unhealthy
          kind: Deployment
          name: team-team-rdhbvc8n
          namespace: team-rdhbvc8n
          status: 'Available: 0/1'
          version: v1
         ....
        resourceCount: 3

plan status with all resources healthy:

status:
    compositionGeneration: 1
    compositionUID: 10785f33-0a4a-41a1-a63a-dfaf9d4aee89
    conditions:
    - lastTransitionTime: "2024-08-06T17:23:20Z"
      message: 'Evaluated and Applied stages: server'
      reason: ProcessedAllStages
      status: "True"
      type: Ready
    generation: 2
    inputGeneration: 1
    stages:
      server:
        appliedCount: 3
        lastApplied:
        - group: apps
          health: Healthy
          kind: Deployment
          name: team-team-h26bql6s
          namespace: team-h26bql6s
          status: 'Deployment is available. Replicas: 1'
          version: v1
        - health: Healthy
          kind: Service
          name: team-team-h26bql6s-landing
          namespace: team-h26bql6s
          status: Service is ready
          version: v1
        - health: Healthy
          kind: ConfigMap
          name: team-team-h26bql6s-page
          namespace: team-h26bql6s
          status: Resource is always ready
          version: v1

barney-s · 2024-08-06T17:27:45Z

/assign @cheftako
/assign @justinsb
/assign @xiaoweim

experiments/compositions/composition/api/v1alpha1/composition_types.go

- apply + wait for ready - Change Applier to get status using CEL rules if rule present - fallback Logic to get status of applied objects using kstatus - Applier.AreResourcesReady() wraps readiness checks - Each stage is applied and then checks if resources are ready (not blocking reconciler) - If resources are not ready, reconciler returns with an error causing a queued reconcile. - CEL based Readiness Rule support - Change Composition spec to take a list of CEL rules for custom status rules - pkg/cel/engine.go - Simple CEL wrapper to run rules against - Testcases: - TestFirstCompositionTeamPage: applies teampage sample (will tests waiting for deployment becoming healthy) - TestCustomStatusCELRule: tests using a CEL readiness rule for Deployment

cheftako · 2024-08-07T21:50:32Z

experiments/compositions/samples/FirstComposition/composition/teampage.yaml

@@ -16,6 +16,7 @@ apiVersion: composition.google.com/v1alpha1
 kind: Composition
 metadata:
 name: team-page
+ namespace: default


Wondering if we should start with the CC structure in which case this would be "config-control".

Can do. But this should work fine in any namespace. Compositions can exist in any namespace that the admin chooses it to be. The Facades can be in any namespace as well.

experiments/compositions/composition/pkg/applier/applier.go

justinsb · 2024-08-07T21:59:01Z

experiments/compositions/composition/pkg/applier/applier.go

+	} else {
+		// Fall back to kstatus
+		result, err := status.Compute(u)
+		if err == nil {


Maybe handle err != nil more clearly here? But I guess we're already handling it acceptably

experiments/compositions/composition/pkg/applier/applier.go

justinsb · 2024-08-07T22:00:55Z

experiments/compositions/composition/pkg/cel/engine.go

+
+func NewEngine(u *unstructured.Unstructured) (*Engine, error) {
+	// TODO: what withtagname ? can we pass yaml ?
+	registry := easycel.NewRegistry("cel-engine", easycel.WithTagName("json"))


We should ask the CEL folk what CEL library they recommend

justinsb · 2024-08-07T22:02:37Z

Generally LGTM. A few comments, the only real blocker I think is whether we need to refetch the object when checking status. I don't think we do; we should have it from the apply.

barney-s · 2024-08-13T17:59:11Z

Generally LGTM. A few comments, the only real blocker I think is whether we need to refetch the object when checking status. I don't think we do; we should have it from the apply.

With the changes in KDP to allow callbacks for computing health, we no longer need to fetch the object. Updated the code (second commit) to register a callback and not fetch anymore.

9197897

google-oss-prow · 2024-08-13T18:20:44Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: justinsb

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [justinsb]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

google-oss-prow bot added the do-not-merge/work-in-progress label Aug 5, 2024

barney-s force-pushed the cel-readiness branch 7 times, most recently from 390569d to 0d2ecf9 Compare August 6, 2024 16:52

barney-s changed the title ~~WIP: using kstatus and cel-rules for readiness checks~~ WIP: Wait for applied objects to become healthy Aug 6, 2024

barney-s force-pushed the cel-readiness branch from 0d2ecf9 to e2b20be Compare August 6, 2024 17:26

barney-s changed the title ~~WIP: Wait for applied objects to become healthy~~ Wait for applied objects to become healthy Aug 6, 2024

google-oss-prow bot removed the do-not-merge/work-in-progress label Aug 6, 2024

google-oss-prow bot assigned cheftako, justinsb and xiaoweim Aug 6, 2024

xiaoweim reviewed Aug 6, 2024

View reviewed changes

experiments/compositions/composition/api/v1alpha1/composition_types.go Show resolved Hide resolved

barney-s force-pushed the cel-readiness branch from e2b20be to 444b44b Compare August 7, 2024 05:39

barney-s force-pushed the cel-readiness branch from 444b44b to 23c356d Compare August 7, 2024 21:40

barney-s changed the title ~~Wait for applied objects to become healthy~~ Wait for applied objects to become healthy using CEL readiness rule or kstatus Aug 7, 2024

cheftako reviewed Aug 7, 2024

View reviewed changes

justinsb reviewed Aug 7, 2024

View reviewed changes

Use newer applylib that supports custom healthcheck callback

9197897

barney-s force-pushed the cel-readiness branch from 6403bfd to 9197897 Compare August 13, 2024 17:44

justinsb approved these changes Aug 13, 2024

View reviewed changes

google-oss-prow bot added the lgtm label Aug 13, 2024

google-oss-prow bot added the approved label Aug 13, 2024

google-oss-prow bot merged commit f5d86cc into GoogleCloudPlatform:master Aug 13, 2024
8 checks passed

barney-s deleted the cel-readiness branch August 29, 2024 21:58

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Wait for applied objects to become healthy using CEL readiness rule or kstatus #2419

Wait for applied objects to become healthy using CEL readiness rule or kstatus #2419

barney-s commented Aug 5, 2024 •

edited

Loading

barney-s commented Aug 6, 2024

cheftako Aug 7, 2024

barney-s Aug 7, 2024

justinsb Aug 7, 2024

justinsb Aug 7, 2024

justinsb commented Aug 7, 2024

barney-s commented Aug 13, 2024 •

edited

Loading

google-oss-prow bot commented Aug 13, 2024

Wait for applied objects to become healthy using CEL readiness rule or kstatus #2419

Wait for applied objects to become healthy using CEL readiness rule or kstatus #2419

Conversation

barney-s commented Aug 5, 2024 • edited Loading

Change description

Sample outputs

CEL rule based checks

kstatus based checks

barney-s commented Aug 6, 2024

cheftako Aug 7, 2024

Choose a reason for hiding this comment

barney-s Aug 7, 2024

Choose a reason for hiding this comment

justinsb Aug 7, 2024

Choose a reason for hiding this comment

justinsb Aug 7, 2024

Choose a reason for hiding this comment

justinsb commented Aug 7, 2024

barney-s commented Aug 13, 2024 • edited Loading

google-oss-prow bot commented Aug 13, 2024

barney-s commented Aug 5, 2024 •

edited

Loading

barney-s commented Aug 13, 2024 •

edited

Loading