Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Splitting functional tests as cloud and non-cloud #7716

Draft
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

ytimocin
Copy link
Contributor

@ytimocin ytimocin commented Jun 24, 2024

Description

The goal of this PR is to create another workflow to run functional tests that don't use cloud resources and also remove the non-cloud tests from the existing functional test workflow that requires approval. In brief:

  1. Workflow: functional-test-cloud (This requires an approval)
  2. Workflow: functional-test-noncloud (This doesn't require an approval and it should be enabled on forks)

RP tests that use cloud resources

  • Core
  • DataStores
  • UCP

RP tests that use non-cloud resources

  • CLI
  • Core
  • Dapr
  • DataStores
  • Kubernetes
  • Messaging
  • Samples
  • UCP

Type of change

@radius-functional-tests
Copy link

radius-functional-tests bot commented Jun 24, 2024

Radius functional test overview

🔍 Go to test action run

Name Value
Repository radius-project/radius
Commit ref be8b05c
Unique ID func650b94b0fb
Image tag pr-func650b94b0fb
Click here to see the list of tools in the current test run
  • gotestsum 1.10.0
  • KinD: v0.20.0
  • Dapr: 1.12.0
  • Azure KeyVault CSI driver: 1.4.2
  • Azure Workload identity webhook: 1.1.0
  • Bicep recipe location ghcr.io/radius-project/dev/test/testrecipes/test-bicep-recipes/<name>:pr-func650b94b0fb
  • Terraform recipe location http://tf-module-server.radius-test-tf-module-server.svc.cluster.local/<name>.zip (in cluster)
  • applications-rp test image location: ghcr.io/radius-project/dev/applications-rp:pr-func650b94b0fb
  • controller test image location: ghcr.io/radius-project/dev/controller:pr-func650b94b0fb
  • ucp test image location: ghcr.io/radius-project/dev/ucpd:pr-func650b94b0fb
  • deployment-engine test image location: ghcr.io/radius-project/deployment-engine:latest

Test Status

⌛ Building Radius and pushing container images for functional tests...
✅ Container images build succeeded
⌛ Publishing Bicep Recipes for functional tests...
✅ Recipe publishing succeeded
⌛ Starting cli functional tests...
⌛ Starting shared functional tests...
⌛ Starting daprrp functional tests...
⌛ Starting datastoresrp functional tests...
⌛ Starting kubernetes functional tests...
⌛ Starting ucp functional tests...
⌛ Starting msgrp functional tests...
⌛ Starting samples functional tests...
❌ Failed to install Radius for cli functional test. Please check the logs for more details
❌ Failed to install Radius for shared functional test. Please check the logs for more details
❌ Failed to install Radius for daprrp functional test. Please check the logs for more details
❌ cli functional test failed. Please check the logs for more details
❌ shared functional test failed. Please check the logs for more details
❌ Failed to install Radius for datastoresrp functional test. Please check the logs for more details
❌ daprrp functional test failed. Please check the logs for more details
❌ datastoresrp functional test failed. Please check the logs for more details
❌ Failed to install Radius for kubernetes functional test. Please check the logs for more details
❌ Failed to install Radius for ucp functional test. Please check the logs for more details
❌ kubernetes functional test cancelled. Please check the logs for more details
❌ samples functional test cancelled. Please check the logs for more details
❌ msgrp functional test cancelled. Please check the logs for more details
❌ ucp functional test cancelled. Please check the logs for more details

Copy link

github-actions bot commented Jun 24, 2024

Unit Tests

3 218 tests  ±0   3 212 ✅ ±0   4m 11s ⏱️ +7s
  261 suites ±0       6 💤 ±0 
    1 files   ±0       0 ❌ ±0 

Results for commit 75f36e2. ± Comparison against base commit 13d50f7.

♻️ This comment has been updated with latest results.

Copy link

codecov bot commented Jun 24, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 61.00%. Comparing base (13d50f7) to head (75f36e2).

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #7716   +/-   ##
=======================================
  Coverage   60.99%   61.00%           
=======================================
  Files         520      520           
  Lines       27010    27010           
=======================================
+ Hits        16476    16478    +2     
+ Misses       9081     9080    -1     
+ Partials     1453     1452    -1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@radius-functional-tests
Copy link

radius-functional-tests bot commented Jun 24, 2024

❌ Container images build failed

❌ Test recipe publishing failed

@radius-functional-tests

This comment has been minimized.

@radius-functional-tests
Copy link

radius-functional-tests bot commented Jun 24, 2024

⌛ Building Radius and pushing container images for functional tests...

✅ Container images build succeeded
⌛ Publishing Bicep Recipes for functional tests...
✅ Recipe publishing succeeded
⌛ Starting samples-noncloud functional tests...
⌛ Starting datastoresrp-noncloud functional tests...
⌛ Starting msgrp-noncloud functional tests...
⌛ Starting kubernetes-noncloud functional tests...
⌛ Starting corerp-noncloud functional tests...
⌛ Starting daprrp-noncloud functional tests...
❌ Failed to install Radius for samples-noncloud functional test. Please check the logs for more details
❌ Failed to install Radius for msgrp-noncloud functional test. Please check the logs for more details
❌ Failed to install Radius for datastoresrp-noncloud functional test. Please check the logs for more details
❌ samples-noncloud functional test failed. Please check the logs for more details
❌ msgrp-noncloud functional test failed. Please check the logs for more details
❌ datastoresrp-noncloud functional test cancelled. Please check the logs for more details
❌ cli-noncloud functional test cancelled. Please check the logs for more details
❌ ucp-noncloud functional test cancelled. Please check the logs for more details
❌ corerp-noncloud functional test cancelled. Please check the logs for more details

@radius-functional-tests
Copy link

radius-functional-tests bot commented Jun 25, 2024

Radius functional test overview

🔍 Go to test action run

Name Value
Repository radius-project/radius
Commit ref 399a3c3
Unique ID func30605840d6
Image tag pr-func30605840d6
Click here to see the list of tools in the current test run
  • gotestsum 1.10.0
  • KinD: v0.20.0
  • Dapr: 1.12.0
  • Azure KeyVault CSI driver: 1.4.2
  • Azure Workload identity webhook: 1.1.0
  • Bicep recipe location ghcr.io/radius-project/dev/test/testrecipes/test-bicep-recipes/<name>:pr-func30605840d6
  • Terraform recipe location http://tf-module-server.radius-test-tf-module-server.svc.cluster.local/<name>.zip (in cluster)
  • applications-rp test image location: ghcr.io/radius-project/dev/applications-rp:pr-func30605840d6
  • controller test image location: ghcr.io/radius-project/dev/controller:pr-func30605840d6
  • ucp test image location: ghcr.io/radius-project/dev/ucpd:pr-func30605840d6
  • deployment-engine test image location: ghcr.io/radius-project/deployment-engine:latest

Test Status

⌛ Building Radius and pushing container images for functional tests...
✅ Container images build succeeded
⌛ Publishing Bicep Recipes for functional tests...
✅ Recipe publishing succeeded
❌ Test tool installation for cli-noncloud failed. Please check the logs for more details
❌ Failed to install Radius for cli-noncloud functional test. Please check the logs for more details
❌ Test tool installation for datastoresrp-noncloud failed. Please check the logs for more details
❌ Failed to install Radius for datastoresrp-noncloud functional test. Please check the logs for more details
❌ Test tool installation for corerp-noncloud failed. Please check the logs for more details
❌ Failed to install Radius for corerp-noncloud functional test. Please check the logs for more details
❌ cli-noncloud functional test failed. Please check the logs for more details
❌ Test tool installation for daprrp-noncloud failed. Please check the logs for more details
❌ datastoresrp-noncloud functional test cancelled. Please check the logs for more details
❌ ucp-noncloud functional test cancelled. Please check the logs for more details
❌ kubernetes-noncloud functional test cancelled. Please check the logs for more details
❌ corerp-noncloud functional test cancelled. Please check the logs for more details
❌ daprrp-noncloud functional test cancelled. Please check the logs for more details
❌ samples-noncloud functional test cancelled. Please check the logs for more details

@radius-functional-tests
Copy link

radius-functional-tests bot commented Jun 25, 2024

Radius functional test overview

🔍 Go to test action run

Name Value
Repository radius-project/radius
Commit ref afefa5b
Unique ID func954c0d57ca
Image tag pr-func954c0d57ca
Click here to see the list of tools in the current test run
  • gotestsum 1.10.0
  • KinD: v0.20.0
  • Dapr: 1.12.0
  • Azure KeyVault CSI driver: 1.4.2
  • Azure Workload identity webhook: 1.1.0
  • Bicep recipe location ghcr.io/radius-project/dev/test/testrecipes/test-bicep-recipes/<name>:pr-func954c0d57ca
  • Terraform recipe location http://tf-module-server.radius-test-tf-module-server.svc.cluster.local/<name>.zip (in cluster)
  • applications-rp test image location: ghcr.io/radius-project/dev/applications-rp:pr-func954c0d57ca
  • controller test image location: ghcr.io/radius-project/dev/controller:pr-func954c0d57ca
  • ucp test image location: ghcr.io/radius-project/dev/ucpd:pr-func954c0d57ca
  • deployment-engine test image location: ghcr.io/radius-project/deployment-engine:latest

Test Status

⌛ Building Radius and pushing container images for functional tests...
✅ Container images build succeeded
⌛ Publishing Bicep Recipes for functional tests...
✅ Recipe publishing succeeded
⌛ Starting msgrp-noncloud functional tests...
⌛ Starting corerp-noncloud functional tests...
⌛ Starting kubernetes-noncloud functional tests...
⌛ Starting datastoresrp-noncloud functional tests...
⌛ Starting cli-noncloud functional tests...
⌛ Starting ucp-noncloud functional tests...
⌛ Starting samples-noncloud functional tests...
❌ Failed to install Radius for msgrp-noncloud functional test. Please check the logs for more details
❌ Failed to install Radius for daprrp-noncloud functional test. Please check the logs for more details
❌ Failed to install Radius for datastoresrp-noncloud functional test. Please check the logs for more details
❌ corerp-noncloud functional test failed. Please check the logs for more details
❌ msgrp-noncloud functional test failed. Please check the logs for more details
❌ daprrp-noncloud functional test failed. Please check the logs for more details
❌ datastoresrp-noncloud functional test failed. Please check the logs for more details
❌ datastoresrp-noncloud functional test cancelled. Please check the logs for more details
❌ cli-noncloud functional test cancelled. Please check the logs for more details
❌ kubernetes-noncloud functional test cancelled. Please check the logs for more details
❌ ucp-noncloud functional test cancelled. Please check the logs for more details
❌ samples-noncloud functional test cancelled. Please check the logs for more details

@radius-functional-tests
Copy link

radius-functional-tests bot commented Jun 25, 2024

Radius functional test overview

🔍 Go to test action run

Name Value
Repository radius-project/radius
Commit ref 273fd42
Unique ID func5f94de1897
Image tag pr-func5f94de1897
Click here to see the list of tools in the current test run
  • gotestsum 1.10.0
  • KinD: v0.20.0
  • Dapr: 1.12.0
  • Azure KeyVault CSI driver: 1.4.2
  • Azure Workload identity webhook: 1.1.0
  • Bicep recipe location ghcr.io/radius-project/dev/test/testrecipes/test-bicep-recipes/<name>:pr-func5f94de1897
  • Terraform recipe location http://tf-module-server.radius-test-tf-module-server.svc.cluster.local/<name>.zip (in cluster)
  • applications-rp test image location: ghcr.io/radius-project/dev/applications-rp:pr-func5f94de1897
  • controller test image location: ghcr.io/radius-project/dev/controller:pr-func5f94de1897
  • ucp test image location: ghcr.io/radius-project/dev/ucpd:pr-func5f94de1897
  • deployment-engine test image location: ghcr.io/radius-project/deployment-engine:latest

Test Status

⌛ Building Radius and pushing container images for functional tests...
✅ Container images build succeeded
⌛ Publishing Bicep Recipes for functional tests...
✅ Recipe publishing succeeded
⌛ Starting msgrp-noncloud functional tests...
⌛ Starting cli-noncloud functional tests...
⌛ Starting daprrp-noncloud functional tests...
⌛ Starting corerp-noncloud functional tests...
⌛ Starting datastoresrp-noncloud functional tests...
⌛ Starting kubernetes-noncloud functional tests...
⌛ Starting samples-noncloud functional tests...
❌ Failed to install Radius for cli-noncloud functional test. Please check the logs for more details
❌ Failed to install Radius for daprrp-noncloud functional test. Please check the logs for more details
❌ Failed to install Radius for datastoresrp-noncloud functional test. Please check the logs for more details
❌ Failed to install Radius for ucp-noncloud functional test. Please check the logs for more details
❌ msgrp-noncloud functional test failed. Please check the logs for more details
❌ daprrp-noncloud functional test failed. Please check the logs for more details
❌ datastoresrp-noncloud functional test cancelled. Please check the logs for more details
❌ corerp-noncloud functional test cancelled. Please check the logs for more details
❌ kubernetes-noncloud functional test cancelled. Please check the logs for more details
❌ samples-noncloud functional test cancelled. Please check the logs for more details

which rad || { echo "cannot find rad"; exit 1; }

echo "*** Installing Radius to Kubernetes ***"
rad install kubernetes \
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Getting this error: *** Installing Radius to Kubernetes *** Installing Radius version afefa5b to namespace: radius-system... Error: failed to run Radius Helm install, err: cannot re-use a name that is still in use Helm output: getting history for release radius creating 1 resource(s) creating 1 resource(s) creating 1 resource(s) beginning wait for 3 resources with timeout of 1m0s.

Link to an example run: https://github.com/radius-project/radius/actions/runs/9659072267/job/26641840926.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Getting the same error.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removing --reinstall.

@radius-functional-tests
Copy link

radius-functional-tests bot commented Jun 25, 2024

Radius functional test overview

🔍 Go to test action run

Name Value
Repository radius-project/radius
Commit ref e6aa7c0
Unique ID func9e6c54c190
Image tag pr-func9e6c54c190
Click here to see the list of tools in the current test run
  • gotestsum 1.10.0
  • KinD: v0.20.0
  • Dapr: 1.12.0
  • Azure KeyVault CSI driver: 1.4.2
  • Azure Workload identity webhook: 1.1.0
  • Bicep recipe location ghcr.io/radius-project/dev/test/testrecipes/test-bicep-recipes/<name>:pr-func9e6c54c190
  • Terraform recipe location http://tf-module-server.radius-test-tf-module-server.svc.cluster.local/<name>.zip (in cluster)
  • applications-rp test image location: ghcr.io/radius-project/dev/applications-rp:pr-func9e6c54c190
  • controller test image location: ghcr.io/radius-project/dev/controller:pr-func9e6c54c190
  • ucp test image location: ghcr.io/radius-project/dev/ucpd:pr-func9e6c54c190
  • deployment-engine test image location: ghcr.io/radius-project/deployment-engine:latest

Test Status

⌛ Building Radius and pushing container images for functional tests...
✅ Container images build succeeded
⌛ Publishing Bicep Recipes for functional tests...
✅ Recipe publishing succeeded
⌛ Starting datastoresrp-noncloud functional tests...
⌛ Starting kubernetes-noncloud functional tests...
⌛ Starting samples-noncloud functional tests...
⌛ Starting msgrp-noncloud functional tests...
⌛ Starting corerp-noncloud functional tests...
⌛ Starting cli-noncloud functional tests...
⌛ Starting ucp-noncloud functional tests...
❌ Failed to install Radius for datastoresrp-noncloud functional test. Please check the logs for more details
❌ Failed to install Radius for msgrp-noncloud functional test. Please check the logs for more details
❌ Failed to install Radius for kubernetes-noncloud functional test. Please check the logs for more details
❌ msgrp-noncloud functional test failed. Please check the logs for more details
❌ daprrp-noncloud functional test cancelled. Please check the logs for more details
❌ kubernetes-noncloud functional test cancelled. Please check the logs for more details
❌ samples-noncloud functional test cancelled. Please check the logs for more details
❌ cli-noncloud functional test cancelled. Please check the logs for more details
❌ corerp-noncloud functional test cancelled. Please check the logs for more details
❌ ucp-noncloud functional test cancelled. Please check the logs for more details

@radius-functional-tests
Copy link

radius-functional-tests bot commented Jun 25, 2024

Radius functional test overview

🔍 Go to test action run

Name Value
Repository radius-project/radius
Commit ref 530192c
Unique ID funcdf250d8c13
Image tag pr-funcdf250d8c13
Click here to see the list of tools in the current test run
  • gotestsum 1.10.0
  • KinD: v0.20.0
  • Dapr: 1.12.0
  • Azure KeyVault CSI driver: 1.4.2
  • Azure Workload identity webhook: 1.1.0
  • Bicep recipe location ghcr.io/radius-project/dev/test/testrecipes/test-bicep-recipes/<name>:pr-funcdf250d8c13
  • Terraform recipe location http://tf-module-server.radius-test-tf-module-server.svc.cluster.local/<name>.zip (in cluster)
  • applications-rp test image location: ghcr.io/radius-project/dev/applications-rp:pr-funcdf250d8c13
  • controller test image location: ghcr.io/radius-project/dev/controller:pr-funcdf250d8c13
  • ucp test image location: ghcr.io/radius-project/dev/ucpd:pr-funcdf250d8c13
  • deployment-engine test image location: ghcr.io/radius-project/deployment-engine:latest

Test Status

⌛ Building Radius and pushing container images for functional tests...
✅ Container images build succeeded
⌛ Publishing Bicep Recipes for functional tests...
✅ Recipe publishing succeeded
⌛ Starting samples-noncloud functional tests...
⌛ Starting msgrp-noncloud functional tests...
⌛ Starting daprrp-noncloud functional tests...
⌛ Starting cli-noncloud functional tests...
⌛ Starting corerp-noncloud functional tests...
⌛ Starting ucp-noncloud functional tests...
⌛ Starting kubernetes-noncloud functional tests...
❌ Failed to install Radius for samples-noncloud functional test. Please check the logs for more details
❌ Failed to install Radius for datastoresrp-noncloud functional test. Please check the logs for more details
❌ Failed to install Radius for cli-noncloud functional test. Please check the logs for more details
❌ Failed to install Radius for daprrp-noncloud functional test. Please check the logs for more details
❌ Failed to install Radius for corerp-noncloud functional test. Please check the logs for more details
❌ samples-noncloud functional test failed. Please check the logs for more details
❌ datastoresrp-noncloud functional test failed. Please check the logs for more details
❌ cli-noncloud functional test failed. Please check the logs for more details
❌ daprrp-noncloud functional test failed. Please check the logs for more details
❌ corerp-noncloud functional test cancelled. Please check the logs for more details
❌ msgrp-noncloud functional test cancelled. Please check the logs for more details
❌ kubernetes-noncloud functional test cancelled. Please check the logs for more details
❌ ucp-noncloud functional test cancelled. Please check the logs for more details

Signed-off-by: ytimocin <ytimocin@microsoft.com>
@radius-functional-tests
Copy link

radius-functional-tests bot commented Jun 25, 2024

Radius functional test (non-cloud) overview

🔍 Go to test action run

Name Value
Repository radius-project/radius
Commit ref 75f36e2
Unique ID funcc3af4faf66
Image tag pr-funcc3af4faf66
Click here to see the list of tools in the current test run
  • gotestsum 1.10.0
  • KinD: v0.20.0
  • Dapr: 1.12.0
  • Azure KeyVault CSI driver: 1.4.2
  • Azure Workload identity webhook: 1.1.0
  • Bicep recipe location ghcr.io/radius-project/dev/test/testrecipes/test-bicep-recipes/<name>:pr-funcc3af4faf66
  • Terraform recipe location http://tf-module-server.radius-test-tf-module-server.svc.cluster.local/<name>.zip (in cluster)
  • applications-rp test image location: ghcr.io/radius-project/dev/applications-rp:pr-funcc3af4faf66
  • controller test image location: ghcr.io/radius-project/dev/controller:pr-funcc3af4faf66
  • ucp test image location: ghcr.io/radius-project/dev/ucpd:pr-funcc3af4faf66
  • deployment-engine test image location: ghcr.io/radius-project/deployment-engine:latest

Test Status

⌛ Building Radius and pushing container images for functional tests...
✅ Container images build succeeded
⌛ Publishing Bicep Recipes for functional tests...
✅ Recipe publishing succeeded
⌛ Starting daprrp-noncloud functional tests...
⌛ Starting datastoresrp-noncloud functional tests...
⌛ Starting corerp-noncloud functional tests...
⌛ Starting samples-noncloud functional tests...
⌛ Starting msgrp-noncloud functional tests...
⌛ Starting kubernetes-noncloud functional tests...
⌛ Starting ucp-noncloud functional tests...
⌛ Starting cli-noncloud functional tests...
❌ Failed to install Radius for daprrp-noncloud functional test. Please check the logs for more details
❌ daprrp-noncloud functional test failed. Please check the logs for more details
❌ datastoresrp-noncloud functional test cancelled. Please check the logs for more details
❌ samples-noncloud functional test cancelled. Please check the logs for more details
❌ msgrp-noncloud functional test cancelled. Please check the logs for more details
❌ kubernetes-noncloud functional test cancelled. Please check the logs for more details
❌ corerp-noncloud functional test cancelled. Please check the logs for more details
❌ cli-noncloud functional test cancelled. Please check the logs for more details
❌ ucp-noncloud functional test cancelled. Please check the logs for more details

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

github workflow changes to not require approval for non-cloud tests
1 participant