feat(queue): add new upsertJobScheduler method

[manast]

This PR introduces a couple of new methods as well as new terminology for our "Repeatable Jobs".
Before this PR we had the confusing notion of "repeatable job", where there was also a "meta repeatable job" that stores the actual information on how often a job should be repeated.

With this PR however we aim to make it clearer and easier to use. We introduce the concept of a Job Scheduler, which is nothing else than a job factory for repeatable jobs. The Job Scheduler exists in Redis in the same key and format as the old meta repeatable job, we are just changing the terminology and introducing new methods that simplify its usage while deprecating the older ones.

The new method is called upsertJobScheduler. Which as the name implies either inserts a new scheduler or updates an existing one. Job Schedulers are all referred via their IDs. So you must choose a unique ID for every scheduler. This simplifies the process of updating and deleting them in the future, as well as to map jobs to their respective job schedulers.

This is the complete signature of the new method:

  /**
   * Upserts a scheduler.
   *
   * A scheduler is a job factory that creates jobs at a given interval.
   * Upserting a scheduler will create a new job scheduler or update an existing one.
   * It will also create the first job based on the repeat options and delayed accordingly.
   *
   * @param key - Unique key for the repeatable job meta.
   * @param repeatOpts - Repeat options
   * @param jobTemplate - Job template. If provided it will be used for all the jobs
   * created by the scheduler.
   *
   * @returns The next job to be scheduled (would normally be in delayed state).
   */
  async upsertJobScheduler(
    jobSchedulerId: NameType,
    repeatOpts: Omit<RepeatOptions, 'key'>,
    jobTemplate?: {
      name?: NameType;
      data?: DataType;
      opts?: Omit<JobsOptions, 'repeat'>;
    },
  ): Promise<Job<T, R, N>

[roggervalf]

what about returning same error code as JobBelongsToJobScheduler that is -8

[roggervalf]

there is a current logic to add a repeatable job without an specific key, this value is generated base on the repeat options. We could preserve that functionality by generating jobSchedulerId if it's not specified

[manast]

That's what I want to avoid actually. With this API the key is mandatory, and when the old API gets deprecated it will be mandatory. Forcing the user to specify a key eliminates most of the pain we had with the old API and also makes the code much simpler and robust.

[roggervalf]

got it, let's do it

[SrdjanCosicPrica]

Hey @manast
With these recent changes, it seems almost like a breaking change for the drain() method.
Our tests started failing after this update because we expected it to drain all delayed jobs in the queue, including those that got created by a repeatable job with .drain(true).
What is the approach to achieve the same result now?

[manast]

@SrdjanCosicPrica If you remove the delayed jobs that belongs to a job scheduler then they will stop repeating. You should remove the job scheduler instead if you need to get rid of the repetitions. We do not consider it a breaking change as this was not supposed to happen.

[SrdjanCosicPrica]

Ok I can understand that.
Back then we assumed this was intended behaviour and we also added a fallback through setInterval which basically added back all repeatables if they for some reason were missing. But reading the context I understand it better now, thanks