Improve ballista performance

[yahoNanJing]

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

To be an efficient distributed execution platform, there're still many aspects we need to improve.

Describe the solution you'd like

• For scheduler, efficient task scheduling
  • Remove Keyspace::QueuedJobs #133 Remove Keyspace::QueuedJobs #134
  • Spawn a thread for execution plan generation #131 Spawn a thread for execution plan generation #135
  • Normalize the serialization and deserialization places of protobuf structs #137 Normalize the serialization and deserialization places of protobuf structs #147
  • A stage may act as the input of multiple stages #144 Use a vector to indicate the output links for ExecutionStage #145
  • Ignore the previous job_id inside fill_reservations() #138 Ignore the previous job_id inside fill_reservations() #141
  • Better encapsulation for ExecutionGraph #149 Move ExecutionGraph encoding and decoding logic into execution_graph for better encapsulation #150
  • Introduce CuratorTaskManager for make an active job be curated by only one scheduler #130 Introduce CuratorTaskManager for make an active job be curated by only one scheduler #153
  • Update task status to the task job curator scheduler #179 Update task status to the its job curator scheduler #181
  • Remove revive offer event loop #136 Remove revive offer event loop #156
  • Make executor fetch shuffle partition data in parallel #208 Make fetch shuffle partition data in parallel #256
  • Add LaunchMultiTask rpc interface for executor #209 Add LaunchMultiTask rpc interface for executor #255
  • Remove active execution graph when the related job is successful or failed. #391 Remove active execution graph when the related job is successful or failed #392
  • Save encoded execution plan in the ExecutionStage to reduce cost of task serialization and deserialization #142 Cache encoded stage plan #393
  • Improve launch task efficiency by calling LaunchMultiTask #389 Improve launch task efficiency by calling LaunchMultiTask #394
  • Add round robin executor slots reservation policy for the scheduler to evenly assign tasks to executors #371 Add round robin executor slots reservation policy for the scheduler to evenly assign tasks to executors #395
  • Redefine the executor task slots #132 Add RoundRobinLocal slots policy for caching executor data to avoid seld persistency #396
  • Make the scheduler event loop buffer size configurable #397 Make the scheduler event loop buffer size configurable #398
  • Initial query stages read parquet files and repartition them needlessly #243

Describe alternatives you've considered

Additional context

[andygrove]

I added #243

[yahoNanJing]

With all the PRs merged, we can achieve a relatively high throughput in RoundRobinLocal SlotsPolicy. For a 200ms query with 200 concurrency running 5 minutes, the average query latency is also around 200ms, which proves the task scheduling is very efficient and should not be a bottleneck in a high throughput system.

And for a large query with with more than 2000 tasks in the first stage, the query latency is improved from more than 10s to around 2.5s.

Hi @thinkharderdev and @andygrove, if you have time, you can also help verify on it.

[yahoNanJing]

Since Dan may not have time for reviewing on these PRs, Andy could you at other guys who may be interested on these PRs? Thanks very much.