Scalability - support for read from a specific partition

[ytaous]

follow up on #220

We need this feature to scale. Here's valid case: there's limit for each partition on each consumer group. If there are more than 5 concurrent jobs in a spark app, we are getting the exception - "Exceeded the maximum number of allowed receivers per partition in a consumer group which is 5."
Since we can't add more concurrent jobs, in order to perform faster, the solution is to create more consumer groups, and create multiple spark apps, each would hit different consumer group on specific partition(s). For example, a 32 partitions of Eventhub (1 default consumer group) read by 5 concurrent spark jobs would be much slower than 16 consumer groups, each assigned 2 partitions and read by 1 Spark app. So we can have 16 Spark apps, each has 5 concurrent jobs - total would be 80 concurrent jobs. 16x faster !!

[sabeegrewal]

Hey @ytaous - we talked offline and agreed a more formal proposal is in order. Feel free to share that here!

If you'd like to continue talking about this offline, then feel free to close this issue 👍

[ytaous]

let's talk offline, thanks.