Skip to content

Commit

Permalink
[BEAM-11934] Add enable_file_dynamic_sharding to allow DataflowRunner…
Browse files Browse the repository at this point in the history
… users to opt-in to dynamic sharding logic (apache#17095)

* [BEAM-11934] Remove DataflowRunner override for runner-determined sharding

Since apache#14450, this override artificially constrains dataflow's ability to use autosharding.

* Revert "[BEAM-11934] Remove DataflowRunner override for runner-determined sharding"

This reverts commit f575256.

* change to be an experiment
  • Loading branch information
dpcollins-google committed Mar 17, 2022
1 parent d08cef6 commit 9096f96
Showing 1 changed file with 6 additions and 4 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -523,10 +523,12 @@ private List<PTransformOverride> getOverrides(boolean streaming) {

overridesBuilder.add(KafkaIO.Read.KAFKA_READ_OVERRIDE);

overridesBuilder.add(
PTransformOverride.of(
PTransformMatchers.writeWithRunnerDeterminedSharding(),
new StreamingShardedWriteFactory(options)));
if (!hasExperiment(options, "enable_file_dynamic_sharding")) {
overridesBuilder.add(
PTransformOverride.of(
PTransformMatchers.writeWithRunnerDeterminedSharding(),
new StreamingShardedWriteFactory(options)));
}

overridesBuilder.add(
PTransformOverride.of(
Expand Down

0 comments on commit 9096f96

Please sign in to comment.