generated from kubernetes/kubernetes-template-project
-
Notifications
You must be signed in to change notification settings - Fork 101
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Showing
12 changed files
with
388 additions
and
46 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,83 @@ | ||
# Dry Run Mode | ||
|
||
As your test suits get bigger and complicated over the period of time, it is essential that the toolings used for creating tests provide an easy way to identify and list the tests being | ||
processed as part of your framework when invoked with certain arguments. And this listing needs | ||
to be quick and clean in order to enable quick turn around time of test development. This requirement bring in the need to introduce a `dry-run` behavior into the `e2e-framework`. | ||
|
||
## Unit Of Test | ||
|
||
Go treats each function starting with `Testxxx` as the Test unit. However, the same is not entirely true in case of the `e2e-framework`. This introduces dynamic tests that are generated during the runtime programmatically for each assessment of each feature. | ||
|
||
From the perspective of the `e2e-framework`, the Unit of test is an `Assessment` that actually performs the assertion of an | ||
expected behavior or state of the system. These assessments are run as a sub-test of the main test identified by the function | ||
`Testxxx`. All framework specific behaviors built around this fundamental test unit of `Assessment`. | ||
|
||
## Why not use `test.list` from `go test` ? | ||
|
||
The `test.list` is a great way to run the dry-run equivalent behavior. However, it is not easily extendable into the core of `e2e-framework` as | ||
there are framework specific behavior such as `setup` and `teardown` workflows. | ||
|
||
That, in conjunction with how the `test.list` works, it is not possible to extract information such as the `assessments` in the feature using the `test.list` mode brings the need to introduce a framework specific `dry-run` mode that can work well with `test.list` while providing all | ||
the framework specific benefits of how the Tests to be processed can be listed | ||
|
||
## `--dry-run` mode | ||
`e2e-framework` adds a new CLI flag that can be used while invoking the test called `--dry-run`. This works in conjunction with `test.list` to provide the following behavior. | ||
|
||
1. When the `--dry-run` mode is invoked No Setup/Teardown workflows are processed | ||
2. Will display the Assessments as individual tests like they would be processed if not invoked with `--dry-run` mode | ||
3. Skip all pre-post actions around the Before/After Features or Before/After Tests | ||
|
||
When tests are invoked with `-test.list` argument, the `--dry-run` mode is automatically switched to enabled to make sure setup/teardown as well as the pre-post actions can be skipped. | ||
|
||
## Example Output with `--dry-run` | ||
```bash | ||
❯ go test . -test.v -args --dry-run | ||
=== RUN TestPodBringUp | ||
=== RUN TestPodBringUp/Feature_One | ||
=== RUN TestPodBringUp/Feature_One/Create_Nginx_Deployment_1 | ||
=== RUN TestPodBringUp/Feature_One/Wait_for_Nginx_Deployment_1_to_be_scaled_up | ||
=== RUN TestPodBringUp/Feature_Two | ||
=== RUN TestPodBringUp/Feature_Two/Create_Nginx_Deployment_2 | ||
=== RUN TestPodBringUp/Feature_Two/Wait_for_Nginx_Deployment_2_to_be_scaled_up | ||
--- PASS: TestPodBringUp (0.00s) | ||
--- PASS: TestPodBringUp/Feature_One (0.00s) | ||
--- PASS: TestPodBringUp/Feature_One/Create_Nginx_Deployment_1 (0.00s) | ||
--- PASS: TestPodBringUp/Feature_One/Wait_for_Nginx_Deployment_1_to_be_scaled_up (0.00s) | ||
--- PASS: TestPodBringUp/Feature_Two (0.00s) | ||
--- PASS: TestPodBringUp/Feature_Two/Create_Nginx_Deployment_2 (0.00s) | ||
--- PASS: TestPodBringUp/Feature_Two/Wait_for_Nginx_Deployment_2_to_be_scaled_up (0.00s) | ||
PASS | ||
ok sigs.k8s.io/e2e-framework/examples/parallel_features 0.353s | ||
``` | ||
|
||
```bash | ||
❯ go test . -test.v -args --dry-run --assess "Deployment 1" | ||
=== RUN TestPodBringUp | ||
=== RUN TestPodBringUp/Feature_One | ||
=== RUN TestPodBringUp/Feature_One/Create_Nginx_Deployment_1 | ||
=== RUN TestPodBringUp/Feature_One/Wait_for_Nginx_Deployment_1_to_be_scaled_up | ||
=== RUN TestPodBringUp/Feature_Two | ||
=== RUN TestPodBringUp/Feature_Two/Create_Nginx_Deployment_2 | ||
env.go:425: Skipping assessment "Create Nginx Deployment 2": name not matched | ||
=== RUN TestPodBringUp/Feature_Two/Wait_for_Nginx_Deployment_2_to_be_scaled_up | ||
env.go:425: Skipping assessment "Wait for Nginx Deployment 2 to be scaled up": name not matched | ||
--- PASS: TestPodBringUp (0.00s) | ||
--- PASS: TestPodBringUp/Feature_One (0.00s) | ||
--- PASS: TestPodBringUp/Feature_One/Create_Nginx_Deployment_1 (0.00s) | ||
--- PASS: TestPodBringUp/Feature_One/Wait_for_Nginx_Deployment_1_to_be_scaled_up (0.00s) | ||
--- PASS: TestPodBringUp/Feature_Two (0.00s) | ||
--- SKIP: TestPodBringUp/Feature_Two/Create_Nginx_Deployment_2 (0.00s) | ||
--- SKIP: TestPodBringUp/Feature_Two/Wait_for_Nginx_Deployment_2_to_be_scaled_up (0.00s) | ||
PASS | ||
ok sigs.k8s.io/e2e-framework/examples/parallel_features 0.945s | ||
``` | ||
|
||
## Example with `-test.list` | ||
```bash | ||
❯ go test . -test.v -test.list ".*" -args | ||
TestPodBringUp | ||
ok sigs.k8s.io/e2e-framework/examples/parallel_features 0.645s | ||
``` | ||
|
||
As you can see from the above two examples, the output of the two commands are not really the same. Using `--dry-run` gives you a more framework specific behavior of how the tests are going to be processed in comparison to `-test.list` | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,54 @@ | ||
# Dry Run of Test Features | ||
|
||
This directory contains the example of how to run the test features in `dry-run` mode using framework specific flags. | ||
|
||
# Run Tests with flags | ||
|
||
These test cases can be executed using the normal `go test` command by passing the right arguments | ||
|
||
```bash | ||
go test -v . -args --dry-run | ||
``` | ||
|
||
With the output generated as following. | ||
|
||
```bash | ||
=== RUN TestDryRunOne | ||
=== RUN TestDryRunOne/F1 | ||
=== RUN TestDryRunOne/F1/Assessment_One | ||
=== RUN TestDryRunOne/F2 | ||
=== RUN TestDryRunOne/F2/Assessment_One | ||
=== RUN TestDryRunOne/F2/Assessment_Two | ||
--- PASS: TestDryRunOne (0.00s) | ||
--- PASS: TestDryRunOne/F1 (0.00s) | ||
--- PASS: TestDryRunOne/F1/Assessment_One (0.00s) | ||
--- PASS: TestDryRunOne/F2 (0.00s) | ||
--- PASS: TestDryRunOne/F2/Assessment_One (0.00s) | ||
--- PASS: TestDryRunOne/F2/Assessment_Two (0.00s) | ||
=== RUN TestDryRunTwo | ||
=== RUN TestDryRunTwo/F1 | ||
=== RUN TestDryRunTwo/F1/Assessment_One | ||
--- PASS: TestDryRunTwo (0.00s) | ||
--- PASS: TestDryRunTwo/F1 (0.00s) | ||
--- PASS: TestDryRunTwo/F1/Assessment_One (0.00s) | ||
PASS | ||
ok sigs.k8s.io/e2e-framework/examples/dry_run 0.618s | ||
``` | ||
|
||
Without the `--dry-run` mode you will see the additional log `Do not run this when in dry-run mode` getting printed onto your terminal. | ||
|
||
In order to integrate this into the `test.list`, please run the following | ||
|
||
```bash | ||
go test -v -list . | ||
``` | ||
|
||
Which generates the output as following | ||
|
||
```bash | ||
TestDryRunOne | ||
TestDryRunTwo | ||
ok sigs.k8s.io/e2e-framework/examples/dry_run 0.375s | ||
``` | ||
|
||
To understand the difference in Output, please refer to the [Design Document](../../docs/design/dry-run-mode.md) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,60 @@ | ||
/* | ||
Copyright 2021 The Kubernetes Authors. | ||
Licensed under the Apache License, Version 2.0 (the "License"); | ||
you may not use this file except in compliance with the License. | ||
You may obtain a copy of the License at | ||
http://www.apache.org/licenses/LICENSE-2.0 | ||
Unless required by applicable law or agreed to in writing, software | ||
distributed under the License is distributed on an "AS IS" BASIS, | ||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
See the License for the specific language governing permissions and | ||
limitations under the License. | ||
*/ | ||
|
||
package dry_run | ||
|
||
import ( | ||
"context" | ||
"testing" | ||
|
||
"k8s.io/klog/v2" | ||
"sigs.k8s.io/e2e-framework/pkg/envconf" | ||
"sigs.k8s.io/e2e-framework/pkg/features" | ||
) | ||
|
||
func TestDryRunOne(t *testing.T) { | ||
f1 := features.New("F1"). | ||
Assess("Assessment One", func(ctx context.Context, t *testing.T, c *envconf.Config) context.Context { | ||
// Perform Some assessment | ||
return ctx | ||
}).Feature() | ||
|
||
f2 := features.New("F2"). | ||
Setup(func(ctx context.Context, t *testing.T, c *envconf.Config) context.Context { | ||
klog.Info("Do not run this when in dry-run mode") | ||
return ctx | ||
}). | ||
Assess("Assessment One", func(ctx context.Context, t *testing.T, c *envconf.Config) context.Context { | ||
// Perform Some assessment | ||
return ctx | ||
}). | ||
Assess("Assessment Two", func(ctx context.Context, t *testing.T, c *envconf.Config) context.Context { | ||
// Perform Some assessment | ||
return ctx | ||
}).Feature() | ||
|
||
testEnv.TestInParallel(t, f1, f2) | ||
} | ||
|
||
func TestDryRunTwo(t *testing.T) { | ||
f1 := features.New("F1"). | ||
Assess("Assessment One", func(ctx context.Context, t *testing.T, c *envconf.Config) context.Context { | ||
// Perform Some assessment | ||
return ctx | ||
}).Feature() | ||
|
||
testEnv.Test(t, f1) | ||
} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,36 @@ | ||
/* | ||
Copyright 2021 The Kubernetes Authors. | ||
Licensed under the Apache License, Version 2.0 (the "License"); | ||
you may not use this file except in compliance with the License. | ||
You may obtain a copy of the License at | ||
http://www.apache.org/licenses/LICENSE-2.0 | ||
Unless required by applicable law or agreed to in writing, software | ||
distributed under the License is distributed on an "AS IS" BASIS, | ||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
See the License for the specific language governing permissions and | ||
limitations under the License. | ||
*/ | ||
|
||
package dry_run | ||
|
||
import ( | ||
"os" | ||
"testing" | ||
|
||
"sigs.k8s.io/e2e-framework/pkg/env" | ||
"sigs.k8s.io/e2e-framework/pkg/envconf" | ||
) | ||
|
||
var ( | ||
testEnv env.Environment | ||
) | ||
|
||
func TestMain(m *testing.M) { | ||
cfg, _ := envconf.NewFromFlags() | ||
testEnv = env.NewWithConfig(cfg) | ||
|
||
os.Exit(testEnv.Run(m)) | ||
} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.