Improve submission speed for many recipes

[jbusecke]

Over at https://github.com/leap-stc/cmip6-leap-feedstock I am picking up the pace and am currently running through several hundreds of datasets a day.

I find that the main limitation for scaling this at the moment is -runner submitting jobs in serial.

I am running submission from here and see a submitted job about every ~30 seconds:

Is there any way to increase the submission speed here? Could we submit batches of jobs at the same time?

[yuvipanda]

We will probably have to implement a threadpool here to make concurrent submission work:

pangeo-forge-runner/pangeo_forge_runner/commands/bake.py

Line 223 in 35e7732

for name, recipe in recipes.items():

[jbusecke]

And then maybe pass an option to limit concurrency? That would be ideal to not fail out due to quota issues.

[yuvipanda]

@jbusecke correct! I think concurrency should just be 1 by default, as that's the most common use case.