Skip to content

Commit

Permalink
Add notation accessory copy test case
Browse files Browse the repository at this point in the history 
Fix goharbor#19546

Signed-off-by: Yang Jiao <jiaoya@vmware.com>
  • Loading branch information
Yang Jiao committed Nov 9, 2023
1 parent 4d2a8fb commit e5465f9
Show file tree
Hide file tree
Showing 5 changed files with 95 additions and 43 deletions.
22 changes: 19 additions & 3 deletions tests/resources/Harbor-Pages/Project.robot
View file
Original file line number Diff line number Diff line change
Expand Up @@ -375,13 +375,29 @@ Back Project Home
[Arguments] ${project_name}
Retry Link Click //a[contains(.,'${project_name}')]

Should Not Be Signed By Cosign
Should Be Signed
[Arguments] ${tag}
Retry Wait Element Visible //clr-dg-row[contains(.,'${tag}')]//clr-icon[contains(@class,'signed')]

Should Not Be Signed
[Arguments] ${tag}
Retry Wait Element Visible //clr-dg-row[contains(.,'${tag}')]//clr-icon[contains(@class,'color-red')]

Should Be Signed By Cosign
[Arguments] ${tag}
Retry Wait Element Visible //clr-dg-row[contains(.,'${tag}')]//clr-icon[contains(@class,'signed')]
[Arguments] ${tag}=${null} ${digest}=${null}
IF '${tag}' != '${null}'
Retry Wait Element Visible //clr-dg-row[./clr-expandable-animation/div/div/div/clr-dg-cell/div/clr-tooltip/div/div/span[contains(.,'${tag}')] and .//clr-dg-row[.//img[@title='signature.cosign']]]
ELSE
Retry Wait Element Visible //clr-dg-row[./clr-expandable-animation/div/div/div/clr-dg-cell/div/a[contains(.,'${digest}')] and .//clr-dg-row[.//img[@title='signature.cosign']]]
END

Should Be Signed By Notation
[Arguments] ${tag}=${null} ${digest}=${null}
IF '${tag}' != '${null}'
Retry Wait Element Visible //clr-dg-row[./clr-expandable-animation/div/div/div/clr-dg-cell/div/clr-tooltip/div/div/span[contains(.,'${tag}')] and .//clr-dg-row[.//img[@title='signature.notation']]]
ELSE
Retry Wait Element Visible //clr-dg-row[./clr-expandable-animation/div/div/div/clr-dg-cell/div/a[contains(.,'${digest}')] and .//clr-dg-row[.//img[@title='signature.notation']]]
END

Delete Accessory
[Arguments] ${tag}
Expand Down
26 changes: 26 additions & 0 deletions tests/resources/Notation_Util.robot
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
# Copyright Project Harbor Authors
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License

*** Settings ***
Documentation This resource provides helper functions for docker operations
Library OperatingSystem
Library Process

*** Keywords ***
Notation Generate Cert
Run And Return Rc And Output notation cert generate-test --default wabbit-networks.io

Notation Sign
[Arguments] ${artifact}
Wait Unitl Command Success notation sign -d --allow-referrers-api ${artifact}
1 change: 1 addition & 0 deletions tests/resources/Util.robot
View file
Original file line number Diff line number Diff line change
Expand Up @@ -82,6 +82,7 @@ Resource SeleniumUtil.robot
Resource Nightly-Util.robot
Resource APITest-Util.robot
Resource Cosign_Util.robot
Resource Notation_Util.robot
Resource Imgpkg-Util.robot
Resource Webhook-Util.robot
Resource TestCaseBody.robot
Expand Down
17 changes: 13 additions & 4 deletions tests/robot-cases/Group1-Nightly/Common.robot
View file
Original file line number Diff line number Diff line change
Expand Up @@ -474,19 +474,28 @@ Test Case - Copy A Image And Accessory
Create An New Project And Go Into Project ${source_project}

Push Image With Tag ${ip} ${user} ${pwd} ${source_project} ${image} ${tag}
Cosign Generate Key Pair
Docker Login ${ip} ${user} ${pwd}
Cosign Generate Key Pair
Cosign Sign ${ip}/${source_project}/${image}:${tag}
Docker Logout ${ip}
Notation Generate Cert
Notation Sign ${ip}/${source_project}/${image}:${tag}

Go Into Repo ${source_project} ${image}
Should Be Signed ${tag}
Retry Button Click ${artifact_list_accessory_btn}
Should Be Signed By Cosign ${tag}
Should Be Signed By Notation ${tag}

Copy Image ${tag} ${target_project} ${image}

Retry Double Keywords When Error Go Into Project ${target_project} Retry Wait Until Page Contains ${image}
Go Into Repo ${target_project} ${image}
Retry Wait Until Page Contains Element //clr-dg-row[contains(.,${tag})]
Should Be Signed ${tag}
Retry Button Click ${artifact_list_accessory_btn}
Should Be Signed By Cosign ${tag}
Should Be Signed By Notation ${tag}
Docker Logout ${ip}
Close Browser

Test Case - Create An New Project With Quotas Set
Expand Down Expand Up @@ -772,14 +781,14 @@ Test Case - Cosign And Cosign Deployment Security Policy
Push Image With Tag ${ip} ${user} ${pwd} project${d} ${image} ${tag}
Go Into Project project${d}
Go Into Repo project${d} ${image}
Should Not Be Signed By Cosign ${tag}
Should Not Be Signed ${tag}
Cannot Pull Image ${ip} ${user} ${pwd} project${d} ${image}:${tag} err_msg=The image is not signed by cosign.
Cosign Generate Key Pair
Cosign Verify ${ip}/project${d}/${image}:${tag} ${false}

Cosign Sign ${ip}/project${d}/${image}:${tag}
Cosign Verify ${ip}/project${d}/${image}:${tag} ${true}
Retry Double Keywords When Error Retry Element Click ${artifact_list_refresh_btn} Should Be Signed By Cosign ${tag}
Retry Double Keywords When Error Retry Element Click ${artifact_list_refresh_btn} Should Be Signed ${tag}
Pull image ${ip} ${user} ${pwd} project${d} ${image}:${tag}

Retry Double Keywords When Error Delete Accessory ${tag} Should be Accessory deleted ${tag}
Expand Down
72 changes: 36 additions & 36 deletions tests/robot-cases/Group1-Nightly/Replication.robot
View file
Original file line number Diff line number Diff line change
Expand Up @@ -389,16 +389,16 @@ Test Case - Robot Account Do Replication
Logout Harbor
Sign In Harbor https://${ip1} ${HARBOR_ADMIN} ${HARBOR_PASSWORD}
Image Should Be Replicated To Project project_dest${d} ${image1}
Should Be Signed By Cosign ${tag1}
Should Be Signed ${tag1}
Image Should Be Replicated To Project project_dest${d} ${image2}
Should Be Signed By Cosign ${tag2}
Should Be Signed ${tag2}
Back Project Home project_dest${d}
Go Into Repo project_dest${d} ${index}
Should Be Signed By Cosign ${index_tag}
Should Be Signed ${index_tag}
Go Into Repo project_dest${d} ${index}
Retry Double Keywords When Error Click Index Achieve ${index_tag} Should Be Signed By Cosign ${image1_short_sha256}
Retry Double Keywords When Error Click Index Achieve ${index_tag} Should Be Signed ${image1_short_sha256}
Go Into Repo project_dest${d} ${index}
Retry Double Keywords When Error Click Index Achieve ${index_tag} Should Not Be Signed By Cosign ${image2_short_sha256}
Retry Double Keywords When Error Click Index Achieve ${index_tag} Should Not Be Signed ${image2_short_sha256}
# pull mode
Logout Harbor
Sign In Harbor ${HARBOR_URL} ${HARBOR_ADMIN} ${HARBOR_PASSWORD}
Expand All @@ -409,16 +409,16 @@ Test Case - Robot Account Do Replication
Check Latest Replication Job Status Succeeded
Check Latest Replication Enabled Copy By Chunk
Image Should Be Replicated To Project project_dest${d} ${image1}
Should Be Signed By Cosign ${tag1}
Should Be Signed ${tag1}
Image Should Be Replicated To Project project_dest${d} ${image2}
Should Be Signed By Cosign ${tag2}
Should Be Signed ${tag2}
Back Project Home project_dest${d}
Go Into Repo project_dest${d} ${index}
Should Be Signed By Cosign ${index_tag}
Should Be Signed ${index_tag}
Go Into Repo project_dest${d} ${index}
Retry Double Keywords When Error Click Index Achieve ${index_tag} Should Be Signed By Cosign ${image1_short_sha256}
Retry Double Keywords When Error Click Index Achieve ${index_tag} Should Be Signed ${image1_short_sha256}
Go Into Repo project_dest${d} ${index}
Retry Double Keywords When Error Click Index Achieve ${index_tag} Should Not Be Signed By Cosign ${image2_short_sha256}
Retry Double Keywords When Error Click Index Achieve ${index_tag} Should Not Be Signed ${image2_short_sha256}
Close Browser

Test Case - Replication Triggered By Events
Expand Down Expand Up @@ -468,28 +468,28 @@ Test Case - Replication Triggered By Events
Logout Harbor
Sign In Harbor ${HARBOR_URL} ${HARBOR_ADMIN} ${HARBOR_PASSWORD}
Go Into Repo project${d} ${image1}
Should Be Signed By Cosign ${tag1}
Should Be Signed ${tag1}
Go Into Repo project${d} ${index}
Should Be Signed By Cosign ${index_tag}
Should Be Signed ${index_tag}
Go Into Repo project${d} ${index}
Retry Double Keywords When Error Click Index Achieve ${index_tag} Should Be Signed By Cosign ${image1_short_sha256}
Retry Double Keywords When Error Click Index Achieve ${index_tag} Should Be Signed ${image1_short_sha256}
Go Into Repo project${d} ${image2}
Should Not Be Signed By Cosign ${tag2}
Should Not Be Signed ${tag2}
Go Into Repo project${d} ${index}
Retry Double Keywords When Error Click Index Achieve ${index_tag} Should Not Be Signed By Cosign ${image2_short_sha256}
Retry Double Keywords When Error Click Index Achieve ${index_tag} Should Not Be Signed ${image2_short_sha256}
Logout Harbor

Sign In Harbor https://${ip1} ${HARBOR_ADMIN} ${HARBOR_PASSWORD}
Go Into Repo project_dest${d} ${image1}
Should Be Signed By Cosign ${tag1}
Should Be Signed ${tag1}
Go Into Repo project_dest${d} ${index}
Should Be Signed By Cosign ${index_tag}
Should Be Signed ${index_tag}
Go Into Repo project_dest${d} ${index}
Retry Double Keywords When Error Click Index Achieve ${index_tag} Should Be Signed By Cosign ${image1_short_sha256}
Retry Double Keywords When Error Click Index Achieve ${index_tag} Should Be Signed ${image1_short_sha256}
Go Into Repo project_dest${d} ${image2}
Should Not Be Signed By Cosign ${tag2}
Should Not Be Signed ${tag2}
Go Into Repo project_dest${d} ${index}
Retry Double Keywords When Error Click Index Achieve ${index_tag} Should Not Be Signed By Cosign ${image2_short_sha256}
Retry Double Keywords When Error Click Index Achieve ${index_tag} Should Not Be Signed ${image2_short_sha256}
Logout Harbor
# delete
Sign In Harbor ${HARBOR_URL} ${HARBOR_ADMIN} ${HARBOR_PASSWORD}
Expand All @@ -498,26 +498,26 @@ Test Case - Replication Triggered By Events
Repo Not Exist project${d} ${image2}
Go Into Repo project${d} ${image1}
Retry Double Keywords When Error Delete Accessory ${tag1} Should be Accessory deleted ${tag1}
Should Not Be Signed By Cosign ${tag1}
Should Not Be Signed ${tag1}
Go Into Repo project${d} ${index}
Retry Double Keywords When Error Delete Accessory ${index_tag} Should be Accessory deleted ${index_tag}
Should Not Be Signed By Cosign ${index_tag}
Should Not Be Signed ${index_tag}
Click Index Achieve ${index_tag}
Retry Double Keywords When Error Delete Accessory ${image1_short_sha256} Should be Accessory deleted ${image1_short_sha256}
Should Not Be Signed By Cosign ${image1_short_sha256}
Should Not Be Signed ${image1_short_sha256}
Logout Harbor

Sign In Harbor https://${ip1} ${HARBOR_ADMIN} ${HARBOR_PASSWORD}
Go Into Repo project_dest${d} ${image2}
Wait Until Page Contains We couldn't find any artifacts!
Go Into Repo project_dest${d} ${image1}
Should be Accessory deleted ${tag1}
Should Not Be Signed By Cosign ${tag1}
Should Not Be Signed ${tag1}
Go Into Repo project_dest${d} ${index}
Should be Accessory deleted ${index_tag}
Should Not Be Signed By Cosign ${index_tag}
Should Not Be Signed ${index_tag}
Retry Double Keywords When Error Click Index Achieve ${index_tag} Should be Accessory deleted ${image1_short_sha256}
Should Not Be Signed By Cosign ${image1_short_sha256}
Should Not Be Signed ${image1_short_sha256}
Close Browser

Test Case - Enable Replication Of Cosign Deployment Security Policy
Expand Down Expand Up @@ -595,31 +595,31 @@ Test Case - Enable Replication Of Cosign Deployment Security Policy
Repo Exist project_pull_dest${d} ${image2}
Repo Exist project_pull_dest${d} ${index}
Go Into Repo project_pull_dest${d} ${image1}
Should Be Signed By Cosign ${tag1}
Should Be Signed ${tag1}
Go Into Repo project_pull_dest${d} ${image2}
Should Be Signed By Cosign ${tag2}
Should Be Signed ${tag2}
Go Into Repo project_pull_dest${d} ${index}
Should Be Signed By Cosign ${index_tag}
Should Be Signed ${index_tag}
Go Into Repo project_pull_dest${d} ${index}
Retry Double Keywords When Error Click Index Achieve ${index_tag} Should Be Signed By Cosign ${image1_short_sha256}
Retry Double Keywords When Error Click Index Achieve ${index_tag} Should Be Signed ${image1_short_sha256}
Go Into Repo project_pull_dest${d} ${index}
Retry Double Keywords When Error Click Index Achieve ${index_tag} Should Be Signed By Cosign ${image2_short_sha256}
Retry Double Keywords When Error Click Index Achieve ${index_tag} Should Be Signed ${image2_short_sha256}
# check project_push_dest
Go Into Project project_push_dest${d}
Switch To Project Repo
Repo Exist project_push_dest${d} ${image1}
Repo Exist project_push_dest${d} ${image2}
Repo Exist project_push_dest${d} ${index}
Go Into Repo project_push_dest${d} ${image1}
Should Be Signed By Cosign ${tag1}
Should Be Signed ${tag1}
Go Into Repo project_push_dest${d} ${image2}
Should Be Signed By Cosign ${tag2}
Should Be Signed ${tag2}
Go Into Repo project_push_dest${d} ${index}
Should Be Signed By Cosign ${index_tag}
Should Be Signed ${index_tag}
Go Into Repo project_push_dest${d} ${index}
Retry Double Keywords When Error Click Index Achieve ${index_tag} Should Be Signed By Cosign ${image1_short_sha256}
Retry Double Keywords When Error Click Index Achieve ${index_tag} Should Be Signed ${image1_short_sha256}
Go Into Repo project_push_dest${d} ${index}
Retry Double Keywords When Error Click Index Achieve ${index_tag} Should Be Signed By Cosign ${image2_short_sha256}
Retry Double Keywords When Error Click Index Achieve ${index_tag} Should Be Signed ${image2_short_sha256}
Close Browser

Test Case - Carvel Imgpkg Copy To Harbor
Expand Down

0 comments on commit e5465f9

Please sign in to comment.