Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Improve Auto scaler pendingTaskBased provisioning strategy to handle when there are no currently running worker node better #11440

Merged
merged 9 commits into from
Jul 14, 2021
Merged

Improve Auto scaler pendingTaskBased provisioning strategy to handle when there are no currently running worker node better #11440

Changes from 1 commit
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
789ea9c
fix pendingTaskBased
maytasm Jul 13, 2021
b1d132e
fix doc
maytasm Jul 13, 2021
2459031
address comments
maytasm Jul 13, 2021
f3955ce
address comments
maytasm Jul 13, 2021
d6bb9a8
address comments
maytasm Jul 13, 2021
06441b9
address comments
maytasm Jul 14, 2021
f5ed8df
address comments
maytasm Jul 14, 2021
298146c
address comments
maytasm Jul 14, 2021
cf39ec2
address comments
maytasm Jul 14, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
address comments
  • Loading branch information
@maytasm
maytasm committed Jul 13, 2021
commit d6bb9a850451df70947d2b77f620cb7300e69622
10 changes: 5 additions & 5 deletions ...pache/druid/indexing/overlord/autoscaling/PendingTaskBasedWorkerProvisioningStrategy.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -246,11 +246,11 @@ private int getScaleUpNodeCount(
log.info("Min/max workers: %d/%d", minWorkerCount, maxWorkerCount);
final int currValidWorkers = getCurrValidWorkers(workers);

// If there are no worker and workerCapacityFallback config is not set (-1) then spin up minWorkerCount
// If there are no worker and workerCapacityFallback config is not set (-1) or invalid (<= 0), then spin up minWorkerCount
// as we cannot determine the exact capacity here to fulfill the need.
// However, if there are no worker but workerCapacityFallback config is set (>0), then we can
// determine the number of workers needed using workerCapacityFallback config as expected worker capacity
int moreWorkersNeeded = currValidWorkers == 0 && config.getWorkerCapacityFallback() > 0 ? minWorkerCount : getWorkersNeededToAssignTasks(
int moreWorkersNeeded = currValidWorkers == 0 && config.getWorkerCapacityFallback() <= 0 ? minWorkerCount : getWorkersNeededToAssignTasks(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: this line is too long. Please break the line. One example would be

      int moreWorkersNeeded = currValidWorkers == 0 && config.getWorkerCapacityFallback() <= 0
                              ? minWorkerCount
                              : getWorkersNeededToAssignTasks(
                                  remoteTaskRunnerConfig,
                                  workerConfig,
                                  pendingTasks,
                                  workers,
                                  config.getWorkerCapacityFallback()
                              );

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

remoteTaskRunnerConfig,
workerConfig,
pendingTasks,
Expand Down Expand Up @@ -284,7 +284,7 @@ private int getWorkersNeededToAssignTasks(
final DefaultWorkerBehaviorConfig workerConfig,
final Collection<Task> pendingTasks,
final Collection<ImmutableWorkerInfo> workers,
final Integer workerCapacityFallback
final int workerCapacityFallback
)
{
final Collection<ImmutableWorkerInfo> validWorkers = Collections2.filter(
Expand Down Expand Up @@ -445,12 +445,12 @@ private int getCurrValidWorkers(Collection<ImmutableWorkerInfo> workers)
return currValidWorkers;
}

private static int getExpectedWorkerCapacity(final Collection<ImmutableWorkerInfo> workers, final Integer workerCapacityFallback)
private static int getExpectedWorkerCapacity(final Collection<ImmutableWorkerInfo> workers, final int workerCapacityFallback)
{
int size = workers.size();
if (size == 0) {
// No existing workers
if (workerCapacityFallback != null) {
if (workerCapacityFallback > 0) {
// Return workerCapacityFallback if it is set in config
return workerCapacityFallback;
} else {
Expand Down