Skip to content

Commit

Permalink
Rename sparkConsumerParallelism to sparkProcessingParallelism to clar…
Browse files Browse the repository at this point in the history
…ify meaning
  • Loading branch information
miguno committed Sep 30, 2014
1 parent c87891c commit b57c8bb
Showing 1 changed file with 2 additions and 2 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -181,11 +181,11 @@ class KafkaSparkStreamingSpec extends FeatureSpec with Matchers with BeforeAndAf
).map(_._2)
}
val unifiedStream = ssc.union(streams) // Merge the "per-partition" DStreams
val sparkConsumerParallelism = 1 // You'd probably pick a higher value than 1 in production.
val sparkProcessingParallelism = 1 // You'd probably pick a higher value than 1 in production.
// Repartition distributes the received batches of data across specified number of machines in the cluster
// before further processing. Essentially, what we are doing here is to decouple processing parallelism from
// reading parallelism (limited by #partitions).
unifiedStream.repartition(sparkConsumerParallelism)
unifiedStream.repartition(sparkProcessingParallelism)
}

// We use accumulators to track the number of consumed and produced messages across all tasks. Named accumulators
Expand Down

0 comments on commit b57c8bb

Please sign in to comment.