Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Helm acceptance tests #2002

Merged
merged 40 commits into from
May 25, 2021
Merged

Helm acceptance tests #2002

merged 40 commits into from
May 25, 2021

Conversation

ijungmann
Copy link
Contributor

@ijungmann ijungmann commented May 19, 2021 •
edited
Loading

Detailed description:
Add the ability to run acceptance tests to the Helm chart.

  • Add a pod template to run the mirror node acceptance tests
  • Add a secret template to hold the acceptance tests application.yml
  • Update documentation to explain new helm tests.

Which issue(s) this PR fixes:
Fixes #1917

Special notes for your reviewer:

Checklist

  • Documentation added
  • Tests updated

@codecov
Copy link

codecov bot commented May 19, 2021 •
edited
Loading

Codecov Report

Merging #2002 (c630aa6) into master (126751e) will increase coverage by 0.40%.
The diff coverage is 82.66%.

❗ Current head c630aa6 differs from pull request most recent head 34c0149. Consider uploading reports for the commit 34c0149 to get more accurate results
Impacted file tree graph

@@             Coverage Diff              @@
##             master    #2002      +/-   ##
============================================
+ Coverage     87.05%   87.45%   +0.40%     
- Complexity     1744     1817      +73     
============================================
  Files           315      320       +5     
  Lines          7731     8213     +482     
  Branches        740      814      +74     
============================================
+ Hits           6730     7183     +453     
- Misses          772      784      +12     
- Partials        229      246      +17
Impacted Files Coverage Δ
...ra/mirror/monitor/config/MonitorConfiguration.java 0.00% <0.00%> (ø)
.../hedera/mirror/monitor/publish/PublishMetrics.java 0.00% <0.00%> (ø)
...a/mirror/monitor/publish/TransactionPublisher.java 1.06% <0.00%> (-0.08%) ⬇️
...a/mirror/monitor/subscribe/AbstractSubscriber.java 77.14% <ø> (-20.00%) ⬇️
.../mirror/monitor/subscribe/rest/RestSubscriber.java 100.00% <ø> (ø)
...nitor/subscribe/rest/RestSubscriberProperties.java 83.33% <ø> (ø)
hedera-mirror-rosetta/cmd/db.go 0.00% <0.00%> (ø)
hedera-mirror-rosetta/cmd/main.go 0.00% <0.00%> (ø)
hedera-mirror-rosetta/tools/maphelper/maphelper.go 100.00% <ø> (ø)
...ra/mirror/monitor/subscribe/SubscribeResponse.java 66.66% <60.00%> (ø)
... and 82 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5a971ac...34c0149. Read the comment docs.

@ijungmann ijungmann added enhancement Type: New feature P2 process Build or test related tasks labels May 24, 2021
@ijungmann ijungmann added this to the Mirror 0.35.0 milestone May 24, 2021
@ijungmann ijungmann self-assigned this May 24, 2021
Ian Jungmann and others added 16 commits May 24, 2021 15:10
Initial work
f458dad 
Signed-off-by: Ian Jungmann <ian.jungmann@hedera.com>
Fix test Dockerfile pointing to flag that does not exist (#1973)
11b1bb8 
- Change hedera-mirror-test Dockerfile default cucumberFlags from @BalanceCheck to @balancecheck

Signed-off-by: Ian Jungmann <ian.jungmann@hedera.com>
@Nana-EC
Fix service endpoint migration (#1974) (#1978)
ebfbf87 
Port
Migration was failing when 0.33.0-rc1 was deployed to dev cluster.
It seems like there was a historic testnet address book where the ports were not set even though the entries had valid IPs. Likely a deployment issue

- Add a default value for port as -1 going forward
- Update migration to set historic address_book_entry values with no port but valid IP to -1. This leaves the window to address this in a future migration. Note this matches the default value when no IP is set in the proto

Signed-off-by: Nana-EC <nana.essilfie-conduah@hedera.com>
Signed-off-by: Ian Jungmann <ian.jungmann@hedera.com>
@Nana-EC
Signature consensus ratio config (#1950)
6a1a753 
OPS and customers would benefit from the ability to customize the consensusRatio used by the importer to validate signature files.

- Add a `consensusRatio` field to the `CommonDownloaderProperties`
- Move `signatureVerificationMetric` from `Downloader.java` to `NodeSignatureVerifier.java`
- Add a `NOT_FOUND` status to the `signatureVerificationMetric ` metric to capture nodes whose signature files were missing from the downloaded list

Signed-off-by: Nana-EC <nana.essilfie-conduah@hedera.com>
Signed-off-by: Ian Jungmann <ian.jungmann@hedera.com>
@steven-sheehy
Fix using wrapper chart without redis chart enabled (#1980)
2fd45c5 
* Bump postgresql-ha chart version
* Bump redis chart version (lots of breaking changes)
* Change `init_v1.sql` to match the chart database init with separate importer user and reduced permissions
* Fix pgpool admin password not auto generating a random password
* Fix `CreateContainerConfigError` due to missing Redis secret
* Fix Importer configured to use Redis sentinel (Google Redis doesn't support sentinel)

Signed-off-by: Steven Sheehy <steven.sheehy@hedera.com>
Signed-off-by: Ian Jungmann <ian.jungmann@hedera.com>
@dependabot
Bump grpc.version from 1.37.0 to 1.37.1 (#1987)
ff07562 
Bumps `grpc.version` from 1.37.0 to 1.37.1.

Updates `grpc-bom` from 1.37.0 to 1.37.1
- [Release notes](https://github.com/grpc/grpc-java/releases)
- [Commits](grpc/grpc-java@v1.37.0...v1.37.1)

Updates `grpc-netty-shaded` from 1.37.0 to 1.37.1
- [Release notes](https://github.com/grpc/grpc-java/releases)
- [Commits](grpc/grpc-java@v1.37.0...v1.37.1)

Updates `grpc-okhttp` from 1.37.0 to 1.37.1
- [Release notes](https://github.com/grpc/grpc-java/releases)
- [Commits](grpc/grpc-java@v1.37.0...v1.37.1)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Signed-off-by: Ian Jungmann <ian.jungmann@hedera.com>
@dependabot
Bump protobuf-java from 3.16.0 to 3.17.0 (#1991)
70d6b56 
Bumps [protobuf-java](https://github.com/protocolbuffers/protobuf) from 3.16.0 to 3.17.0.
- [Release notes](https://github.com/protocolbuffers/protobuf/releases)
- [Changelog](https://github.com/protocolbuffers/protobuf/blob/master/generate_changelog.py)
- [Commits](protocolbuffers/protobuf@v3.16.0...v3.17.0)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Signed-off-by: Ian Jungmann <ian.jungmann@hedera.com>
@dependabot
Bump frontend-maven-plugin from 1.11.3 to 1.12.0 (#1988)
eb5e521 
Bumps [frontend-maven-plugin](https://github.com/eirslett/frontend-maven-plugin) from 1.11.3 to 1.12.0.
- [Release notes](https://github.com/eirslett/frontend-maven-plugin/releases)
- [Changelog](https://github.com/eirslett/frontend-maven-plugin/blob/master/CHANGELOG.md)
- [Commits](eirslett/frontend-maven-plugin@frontend-plugins-1.11.3...frontend-plugins-1.12.0)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Signed-off-by: Ian Jungmann <ian.jungmann@hedera.com>
@dependabot
Bump software.amazon.awssdk:bom from 2.16.59 to 2.16.63 (#1990)
183f2e4 
Bumps [software.amazon.awssdk:bom](https://github.com/aws/aws-sdk-java-v2) from 2.16.59 to 2.16.63.
- [Release notes](https://github.com/aws/aws-sdk-java-v2/releases)
- [Changelog](https://github.com/aws/aws-sdk-java-v2/blob/master/CHANGELOG.md)
- [Commits](aws/aws-sdk-java-v2@2.16.59...2.16.63)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Signed-off-by: Ian Jungmann <ian.jungmann@hedera.com>
@dependabot
Bump embedded.testcontainers.version from 2.0.7 to 2.0.8 (#1994)
3bbe810 
Bumps `embedded.testcontainers.version` from 2.0.7 to 2.0.8.
Updates `embedded-postgresql` from 2.0.7 to 2.0.8
Updates `embedded-redis` from 2.0.7 to 2.0.8
Updates `embedded-google-pubsub` from 2.0.7 to 2.0.8

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Signed-off-by: Ian Jungmann <ian.jungmann@hedera.com>
@dependabot
Bump jackson-dataformat-msgpack from 0.8.22 to 0.8.24 (#1995)
b6a93d4 
Bumps [jackson-dataformat-msgpack](https://github.com/msgpack/msgpack-java) from 0.8.22 to 0.8.24.
- [Release notes](https://github.com/msgpack/msgpack-java/releases)
- [Changelog](https://github.com/msgpack/msgpack-java/blob/develop/RELEASE_NOTES.md)
- [Commits](msgpack/msgpack-java@0.8.22...v0.8.24)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Signed-off-by: Ian Jungmann <ian.jungmann@hedera.com>
@dependabot
Bump cucumber.version from 6.10.3 to 6.10.4 (#1989)
7ed0efd 
Bumps `cucumber.version` from 6.10.3 to 6.10.4.

Updates `cucumber-java` from 6.10.3 to 6.10.4
- [Release notes](https://github.com/cucumber/cucumber-jvm/releases)
- [Changelog](https://github.com/cucumber/cucumber-jvm/blob/main/CHANGELOG.md)
- [Commits](cucumber/cucumber-jvm@v6.10.3...v6.10.4)

Updates `cucumber-junit` from 6.10.3 to 6.10.4
- [Release notes](https://github.com/cucumber/cucumber-jvm/releases)
- [Changelog](https://github.com/cucumber/cucumber-jvm/blob/main/CHANGELOG.md)
- [Commits](cucumber/cucumber-jvm@v6.10.3...v6.10.4)

Updates `cucumber-junit-platform-engine` from 6.10.3 to 6.10.4
- [Release notes](https://github.com/cucumber/cucumber-jvm/releases)
- [Changelog](https://github.com/cucumber/cucumber-jvm/blob/main/CHANGELOG.md)
- [Commits](cucumber/cucumber-jvm@v6.10.3...v6.10.4)

Updates `cucumber-spring` from 6.10.3 to 6.10.4
- [Release notes](https://github.com/cucumber/cucumber-jvm/releases)
- [Changelog](https://github.com/cucumber/cucumber-jvm/blob/main/CHANGELOG.md)
- [Commits](cucumber/cucumber-jvm@v6.10.3...v6.10.4)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Signed-off-by: Ian Jungmann <ian.jungmann@hedera.com>
@dependabot
Bump mathjs in /hedera-mirror-rest/monitoring/monitor_apis (#1982)
e6ff621 
Bumps [mathjs](https://github.com/josdejong/mathjs) from 9.3.2 to 9.4.0.
- [Release notes](https://github.com/josdejong/mathjs/releases)
- [Changelog](https://github.com/josdejong/mathjs/blob/develop/HISTORY.md)
- [Commits](josdejong/mathjs@v9.3.2...v9.4.0)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Signed-off-by: Ian Jungmann <ian.jungmann@hedera.com>
@dependabot
Bump mathjs from 9.3.2 to 9.4.0 in /hedera-mirror-rest (#1986)
ccdd510 
Bumps [mathjs](https://github.com/josdejong/mathjs) from 9.3.2 to 9.4.0.
- [Release notes](https://github.com/josdejong/mathjs/releases)
- [Changelog](https://github.com/josdejong/mathjs/blob/develop/HISTORY.md)
- [Commits](josdejong/mathjs@v9.3.2...v9.4.0)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Signed-off-by: Ian Jungmann <ian.jungmann@hedera.com>
Add alertmanager inhibit rules (#1925)
1df680a 
*  Add a new alert to each service to check when pods are not ready
*  Add inhibit rules to each values.yaml to logically group alerts together
*  Add new alertmanagerconfig template to handle the new inhibit rules section
*  Update the mirror-node-common chart to read the new config

Signed-off-by: Ian Jungmann <ian.jungmann@hedera.com>
@steven-sheehy
Remove JMeter based performance tests (#1981)
acfccd3 
* Fix acceptance test config overwriting default config
* Fix monitor config documentation not showing hedera.mirror.monitor prefix and confusing users
* Fix test image sometimes failing to run due to not downloading all dependencies during build
* Improve test image creation time and image size (1.5GB to 566MB)
* Remove all JMeter related code

Signed-off-by: Steven Sheehy <steven.sheehy@hedera.com>
Signed-off-by: Ian Jungmann <ian.jungmann@hedera.com>
Ian Jungmann added 9 commits May 24, 2021 15:10
Change cucumber tags to template
a63a8a9 
Signed-off-by: Ian Jungmann <ian.jungmann@hedera.com>
Make env into volume
b04ae23 
Signed-off-by: Ian Jungmann <ian.jungmann@hedera.com>
Fix templates
162878d 
Signed-off-by: Ian Jungmann <ian.jungmann@hedera.com>
More template fixes
4071ea3 
Signed-off-by: Ian Jungmann <ian.jungmann@hedera.com>
Stop the restart
1e634a4 
Signed-off-by: Ian Jungmann <ian.jungmann@hedera.com>
Add values.yaml and README
ac97392 
Signed-off-by: Ian Jungmann <ian.jungmann@hedera.com>
Fix README
6305a3e 
Signed-off-by: Ian Jungmann <ian.jungmann@hedera.com>
Fix README
ce4bc20 
Signed-off-by: Ian Jungmann <ian.jungmann@hedera.com>
Disable test by default
5126fef 
Signed-off-by: Ian Jungmann <ian.jungmann@hedera.com>
Ian Jungmann added 2 commits May 24, 2021 15:13
Merge with master
0315456
Fix README
05f9e27 
Signed-off-by: Ian Jungmann <ian.jungmann@hedera.com>
Reorder values.yaml
1a34249 
Signed-off-by: Ian Jungmann <ian.jungmann@hedera.com>
@ijungmann ijungmann marked this pull request as ready for review May 24, 2021 20:18
@ijungmann ijungmann requested a review from a team May 24, 2021 20:18
Change test pod name to acceptance-test
8ae6a81 
Signed-off-by: Ian Jungmann <ian.jungmann@hedera.com>
Nana-EC
Nana-EC requested changes May 24, 2021
View reviewed changes
Copy link
Contributor

@Nana-EC Nana-EC left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking good, some nits and suggestions

charts/hedera-mirror/values.yaml Outdated
# Do not use use 0.0.2 or 0.0.50 for operator to ensure crypto transfers are not waived
operatorId:
operatorKey:
restBaseUrl: "http://{{ .Release.Name }}-rest/api/v1"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should be rest.baseUrl.
Are the rest checks working with this?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch, I just got lucky that the default was testnet. I tested with this corrected and it works, also tested with a bad endpoint and it failed so I know it's using it now.

charts/hedera-mirror/values.yaml Show resolved Hide resolved
charts/hedera-mirror/templates/tests/pod.yaml Show resolved Hide resolved
Ian Jungmann added 3 commits May 25, 2021 01:57
Fix rest base url
5ed0dd3 
Signed-off-by: Ian Jungmann <ian.jungmann@hedera.com>
Add retrieveAddressBook config
668e54c 
Signed-off-by: Ian Jungmann <ian.jungmann@hedera.com>
Adding comments
19175e3 
Signed-off-by: Ian Jungmann <ian.jungmann@hedera.com>
@ijungmann ijungmann requested a review from Nana-EC May 25, 2021 07:09
steven-sheehy
steven-sheehy requested changes May 25, 2021
View reviewed changes
charts/hedera-mirror/templates/tests/pod.yaml Outdated
spec:
containers:
- name: acceptance-tests
image: "{{ .Values.test.image.repository }}:{{ .Values.test.image.tag | default .Chart.AppVersion }}"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should also use the global.image.tag. You'll need to add a global.image: {} to the values.yaml.

Suggested change
image: "{{ .Values.test.image.repository }}:{{ .Values.test.image.tag | default .Chart.AppVersion }}"
image: "{{ .Values.test.image.repository }}:{{ .Values.global.image.tag | default .Values.test.image.tag | default .Chart.AppVersion }}"

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure, added that.

charts/hedera-mirror/templates/tests/secret.yaml Outdated
annotations:
helm.sh/hook: test
helm.sh/hook-delete-policy: hook-failed,hook-succeeded

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: extra line

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed.

PR Comments
34c0149 
Signed-off-by: Ian Jungmann <ian.jungmann@hedera.com>
@sonarcloud
Copy link

sonarcloud bot commented May 25, 2021

SonarCloud Quality Gate failed.

Bug A 0 Bugs
Vulnerability A 0 Vulnerabilities
Security Hotspot A 0 Security Hotspots
Code Smell A 1 Code Smell

0.0% 0.0% Coverage
3.9% 3.9% Duplication

steven-sheehy
steven-sheehy approved these changes May 25, 2021
View reviewed changes
Copy link
Member

@steven-sheehy steven-sheehy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Nana-EC
Nana-EC approved these changes May 25, 2021
View reviewed changes
Copy link
Contributor

@Nana-EC Nana-EC left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@ijungmann ijungmann merged commit 310f21c into master May 25, 2021
@ijungmann ijungmann deleted the helm_acceptance_tests branch May 25, 2021 17:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@steven-sheehy steven-sheehy steven-sheehy approved these changes

@Nana-EC Nana-EC Nana-EC approved these changes

Assignees

@ijungmann ijungmann

Labels
enhancement Type: New feature P2 process Build or test related tasks
Projects
None yet
Milestone
Mirror 0.35.0
Development

Successfully merging this pull request may close these issues.

Helm acceptance tests
4 participants
@ijungmann @steven-sheehy @Nana-EC @hedera-github-bot