-
Notifications
You must be signed in to change notification settings - Fork 3.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Enable querying entirely cold datasources #16676
Merged
Merged
Changes from 1 commit
Commits
Show all changes
32 commits
Select commit
Hold shift + click to select a range
ca33d8b
Fix build
findingrish 1abba25
Merge branch 'master' of github.com:findingrish/druid
findingrish 033c420
temp changes
findingrish 96e77c1
Coordinator changes to process schema for cold segments
findingrish 1739dce
Changes in the broker metadata cache to fetch used datasources schema
findingrish 6ed0b8b
Refactor DruidCoordinator, broker metadata cache changes to execute r…
findingrish 966f5ac
Fix tests
findingrish d586a8e
Merge remote-tracking branch 'upstream/master' into cold_ds_schema
findingrish b72b822
Update MetadataResourceTest
findingrish ad58f9b
minor changes
findingrish 2944959
Fix refresh condition on Broker, add UTs
findingrish b09df54
Fix test
findingrish 254cea2
Merge remote-tracking branch 'upstream/master' into cold_ds_schema
findingrish 25d23c6
minor code changes, add test
findingrish 7169b6d
revert changes
findingrish 89e2d64
Fix test
findingrish ba380ec
checkstyle
findingrish d3c112c
Update docs
findingrish debcb42
review comments
findingrish 479382c
Doc changes
findingrish 87113ca
Update threshold for logging cold schema processing stats
findingrish 2a66ae1
Minor changes
findingrish 695ba92
add debug logging
findingrish a265cf0
Merge hot and cold schema only while querying datasource schema
findingrish 8182f39
Update tests
findingrish 766082e
Update docs
findingrish 2bea1fb
Merge remote-tracking branch 'upstream/master' into cold_ds_schema
findingrish 8582f5d
Update test
findingrish 7ed2d96
Update tests wiht SegmentReplicationStatus mock
findingrish 47bf4ab
Merge remote-tracking branch 'upstream/master' into cold_ds_schema
findingrish 8fb0a04
Minor change
findingrish 87a0926
Minor change
findingrish File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Doc changes
- Loading branch information
commit 479382ccd9f47b6352deb20a89517c79e1eb46b5
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -99,6 +99,7 @@ public class CoordinatorSegmentMetadataCache extends AbstractSegmentMetadataCach | |
{ | ||
private static final EmittingLogger log = new EmittingLogger(CoordinatorSegmentMetadataCache.class); | ||
private static final Long COLD_SCHEMA_PERIOD_MULTIPLIER = 3L; | ||
private static final Long COLD_SCHEMA_SLOWNESS_THRESHOLD_MILLIS = TimeUnit.MINUTES.toMillis(1); | ||
|
||
private final SegmentMetadataCacheConfig config; | ||
private final ColumnTypeMergePolicy columnTypeMergePolicy; | ||
|
@@ -113,6 +114,8 @@ public class CoordinatorSegmentMetadataCache extends AbstractSegmentMetadataCach | |
private final ConcurrentHashMap<String, DataSourceInformation> coldSchemaTable = new ConcurrentHashMap<>(); | ||
|
||
// Period for cold schema processing thread. This is a multiple of segment polling period. | ||
// Cold schema processing runs slower than the segment poll to save processing cost of all segments. | ||
// The downside is a delay in columns from cold segment reflecting in the datasource schema. | ||
private final long coldSchemaExecPeriodMillis; | ||
private final ScheduledExecutorService coldScehmaExec; | ||
private @Nullable Future<?> cacheExecFuture = null; | ||
|
@@ -603,7 +606,7 @@ protected void coldDatasourceSchemaExec() | |
// remove any stale datasource from the map | ||
coldSchemaTable.keySet().retainAll(dataSources); | ||
|
||
if (stopwatch.millisElapsed() > coldSchemaExecPeriodMillis * 0.80) { | ||
if (stopwatch.millisElapsed() > COLD_SCHEMA_SLOWNESS_THRESHOLD_MILLIS) { | ||
log.info("Cold schema processing was slow, taking [%d] millis. " | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Else for this should be debug. |
||
+ "Processed [%d] datasources, [%d] segments & [%d] datasourceWithColdSegments.", | ||
stopwatch.millisElapsed(), datasources, segments, datasourceWithColdSegments | ||
|
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do we have a test case for this ?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, in
CoordinatorSegmentMetadataCacheTest#testColdDatasourceSchema_verifyStaleDatasourceRemoved
.