Add support for S3 failover #4553
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
steven-sheehy
added
enhancement
Codecov ReportBase: 92.71% // Head: 92.51% // Decreases project coverage by
Additional details and impacted files@@ Coverage Diff @@
## main #4553 +/- ##
============================================
- Coverage 92.71% 92.51% -0.21%
- Complexity 2895 2926 +31
============================================
Files 532 535 +3
Lines 17172 17188 +16
Branches 1805 1810 +5
============================================
- Hits 15921 15901 -20
- Misses 904 940 +36
Partials 347 347
Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here. ☔ View full report at Codecov. |
...orter/src/main/java/com/hedera/mirror/importer/downloader/provider/S3StreamFileProvider.java
Show resolved
Hide resolved
Signed-off-by: Steven Sheehy <steven.sheehy@swirldslabs.com>
Signed-off-by: Steven Sheehy <steven.sheehy@swirldslabs.com>
steven-sheehy
force-pushed
the
54-s3-failover
branch
from
8ade773
to
21779a8
Compare
Signed-off-by: Steven Sheehy <steven.sheehy@swirldslabs.com>
steven-sheehy
commented
Sep 27, 2022
| } | ||
| }) | ||
| .filter(s -> s != null && s.getFileType() == SIGNATURE) | ||
| .collect(groupingBy(StreamFilename::getInstant, maxBy(StreamFilename.EXTENSION_COMPARATOR))); |
There was a problem hiding this comment.
I removed the grouping by instant functionality. This would only occur when the account balance csv and pg.gz file were uploaded simultaneously for a short period of time.
It's fine to download either one as they're both equally valid so we can simplify our logic to just rely on the single layer of grouping by StreamFilename (which uses filename without compression suffix as distinct key) via the multimap. That way we don't pay the cost of nested grouping all files for a temporary window.
Signed-off-by: Steven Sheehy <steven.sheehy@swirldslabs.com>
xin-hedera
requested changes
Sep 29, 2022
...or-importer/src/main/java/com/hedera/mirror/importer/addressbook/AddressBookServiceImpl.java
Outdated
Show resolved
Hide resolved
...ror-importer/src/main/java/com/hedera/mirror/importer/downloader/ConsensusValidatorImpl.java
Show resolved
Hide resolved
...irror-importer/src/main/java/com/hedera/mirror/importer/downloader/DownloaderProperties.java
Outdated
Show resolved
Hide resolved
Signed-off-by: Steven Sheehy <steven.sheehy@swirldslabs.com>
|
Kudos, SonarCloud Quality Gate passed!
|
This was referenced Sep 30, 2022
mgoelswirlds
pushed a commit
that referenced
this pull request
Oct 12, 2022
* Add a ConsensusNode abstraction and use it everywhere instead of node account IDs or calling the address book multiple times per file * Add a connection timeout property and increase it to 5s to avoid occasional connection errors. * Add hedera.mirror.importer.downloader.sources properties that grandfathers legacy hedera.mirror.importer.downloader properties as the primary source in list * Change stream file persistence to write files to a node ID based path * Increase downloader batch size to 100 to improve historical sync speed * Refactor Downloader to push S3 logic into StreamFileProvider abstraction * Replace hedera.mirror.importer.downloader.*.batchSize with generic hedera.mirror.importer.downloader.batchSize * Replace hedera.mirror.importer.downloader.*.threads with generic hedera.mirror.importer.downloader.threads * Remove hedera.mirror.importer.downloader.*.prefix in favor of existing StreamType properties Signed-off-by: Steven Sheehy <steven.sheehy@swirldslabs.com> Signed-off-by: mgoelswirlds <mugdha.goel@swirldslabs.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description:
ConsensusNodeabstraction and use it everywhere instead of node account IDs or calling the address book multiple times per filehedera.mirror.importer.downloader.sourcesproperties that grandfathers legacyhedera.mirror.importer.downloaderproperties as the primary source in listDownloaderto push S3 logic intoStreamFileProviderabstractionhedera.mirror.importer.downloader.*.batchSizewith generichedera.mirror.importer.downloader.batchSizehedera.mirror.importer.downloader.*.threadswith generichedera.mirror.importer.downloader.threadshedera.mirror.importer.downloader.*.prefixin favor of existingStreamTypepropertiesRelated issue(s):
Fixes #54
Fixes #4538
Notes for reviewer:
Checklist