Add reservation option for Slurm

[Petraea]

Hi there,
I know for certain that Slurm can take a full text reservation name - perhaps the other schedulers can use this. But having this optional flag is nice for busier clusters.

[rkdarst]

Don't forget to modify the batch script too - just adding an option doesn't do anything. There are examples directly below you can copy and paste.

But I have also been wondered about how many of these options are needed. Right now, you can already do req_options = "--reservation=NAME" (and really put arbitrary options there). We have many hard-coded options which are essentially reduced to that. Unless this data is used somehow, or people deploy with same options across different spawners, I wonder how many are needed.

The above paragraph is more for maintainers than you - just thinking aloud. I don't see any reason to not take this (I'm not a maintainer though).

[Petraea]

Hi there,
I can appreciate the sentiment, but one can also set the partition, the QoS and the account via the req_options. The downside of course to that is that you have to expose raw scheduler information to a potentially novice user, introducing the risk of typos, confusion, etc.
Finding a more elegant method of drawing these options out of the current Slurm version such that they can be presented might not be a bad idea. Something that parses sbatch -u, perhaps?

[rkdarst]

I'm not disagreeing with you... just some random thoughts.

Anyway, the system seems to be add anything that people find useful, and we can organize later. So, I'd say add it to the batch script and I'd call this good.

[rkdarst]

I added to batch script in rkdarst/slurm_reservation, but didn't make a new PR. Instead I integrated to my dev branch and once that is pulled, this should be taken into effect.