-
Notifications
You must be signed in to change notification settings - Fork 390
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
fix: do not remove delayed jobs that belongs to a job scheduler when …
…using drain and clean
- Loading branch information
Showing
21 changed files
with
624 additions
and
205 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,4 +1,4 @@ | ||
| #!/usr/bin/env sh | ||
| . "$(dirname -- "$0")/_/husky.sh" | ||
|
|
||
| npx npm-run-all pretty:quick lint:staged | ||
| yarn npm-run-all pretty:quick lint:staged |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,131 +1,130 @@ | ||
| # Table of contents | ||
|
|
||
| * [What is BullMQ](README.md) | ||
| * [Quick Start](<README (1).md>) | ||
| * [API Reference](https://api.docs.bullmq.io) | ||
| * [Changelogs](changelog.md) | ||
| * [v4](changelogs/changelog-v4.md) | ||
| * [v3](changelogs/changelog-v3.md) | ||
| * [v2](changelogs/changelog-v2.md) | ||
| * [v1](changelogs/changelog-v1.md) | ||
| - [What is BullMQ](README.md) | ||
| - [Quick Start](<README (1).md>) | ||
| - [API Reference](https://api.docs.bullmq.io) | ||
| - [Changelogs](changelog.md) | ||
| - [v4](changelogs/changelog-v4.md) | ||
| - [v3](changelogs/changelog-v3.md) | ||
| - [v2](changelogs/changelog-v2.md) | ||
| - [v1](changelogs/changelog-v1.md) | ||
|
|
||
| ## Guide | ||
|
|
||
| * [Introduction](guide/introduction.md) | ||
| * [Connections](guide/connections.md) | ||
| * [Queues](guide/queues/README.md) | ||
| * [Auto-removal of jobs](guide/queues/auto-removal-of-jobs.md) | ||
| * [Adding jobs in bulk](guide/queues/adding-bulks.md) | ||
| * [Global Concurrency](guide/queues/global-concurrency.md) | ||
| * [Removing Jobs](guide/queues/removing-jobs.md) | ||
| * [Workers](guide/workers/README.md) | ||
| * [Auto-removal of jobs](guide/workers/auto-removal-of-jobs.md) | ||
| * [Concurrency](guide/workers/concurrency.md) | ||
| * [Graceful shutdown](guide/workers/graceful-shutdown.md) | ||
| * [Stalled Jobs](guide/workers/stalled-jobs.md) | ||
| * [Sandboxed processors](guide/workers/sandboxed-processors.md) | ||
| * [Pausing queues](guide/workers/pausing-queues.md) | ||
| * [Jobs](guide/jobs/README.md) | ||
| * [FIFO](guide/jobs/fifo.md) | ||
| * [LIFO](guide/jobs/lifo.md) | ||
| * [Job Ids](guide/jobs/job-ids.md) | ||
| * [Job Data](guide/jobs/job-data.md) | ||
| * [Debouncing](guide/jobs/debouncing.md) | ||
| * [Delayed](guide/jobs/delayed.md) | ||
| * [Repeatable](guide/jobs/repeatable.md) | ||
| * [Prioritized](guide/jobs/prioritized.md) | ||
| * [Removing jobs](guide/jobs/removing-job.md) | ||
| * [Stalled](guide/jobs/stalled.md) | ||
| * [Getters](guide/jobs/getters.md) | ||
| * [Flows](guide/flows/README.md) | ||
| * [Adding flows in bulk](guide/flows/adding-bulks.md) | ||
| * [Get Flow Tree](guide/flows/get-flow-tree.md) | ||
| * [Fail Parent](guide/flows/fail-parent.md) | ||
| * [Remove Dependency](guide/flows/remove-dependency.md) | ||
| * [Ignore Dependency](guide/flows/ignore-dependency.md) | ||
| * [Remove Child Dependency](guide/flows/remove-child-dependency.md) | ||
| * [Metrics](guide/metrics/metrics.md) | ||
| * [Rate limiting](guide/rate-limiting.md) | ||
| * [Parallelism and Concurrency](guide/parallelism-and-concurrency.md) | ||
| * [Retrying failing jobs](guide/retrying-failing-jobs.md) | ||
| * [Returning job data](guide/returning-job-data.md) | ||
| * [Events](guide/events.md) | ||
| * [QueueScheduler](guide/queuescheduler.md) | ||
| * [Redis™ Compatibility](guide/redis-tm-compatibility/README.md) | ||
| * [Dragonfly](guide/redis-tm-compatibility/dragonfly.md) | ||
| * [Redis™ hosting](guide/redis-tm-hosting/README.md) | ||
| * [AWS MemoryDB](guide/redis-tm-hosting/aws-memorydb.md) | ||
| * [AWS Elasticache](guide/redis-tm-hosting/aws-elasticache.md) | ||
| * [Architecture](guide/architecture.md) | ||
| * [NestJs](guide/nestjs/README.md) | ||
| * [Producers](guide/nestjs/producers.md) | ||
| * [Queue Events Listeners](guide/nestjs/queue-events-listeners.md) | ||
| * [Going to production](guide/going-to-production.md) | ||
| * [Migration to newer versions](guide/migration-to-newer-versions.md) | ||
| * [Troubleshooting](guide/troubleshooting.md) | ||
| - [Introduction](guide/introduction.md) | ||
| - [Connections](guide/connections.md) | ||
| - [Queues](guide/queues/README.md) | ||
| - [Auto-removal of jobs](guide/queues/auto-removal-of-jobs.md) | ||
| - [Adding jobs in bulk](guide/queues/adding-bulks.md) | ||
| - [Global Concurrency](guide/queues/global-concurrency.md) | ||
| - [Removing Jobs](guide/queues/removing-jobs.md) | ||
| - [Workers](guide/workers/README.md) | ||
| - [Auto-removal of jobs](guide/workers/auto-removal-of-jobs.md) | ||
| - [Concurrency](guide/workers/concurrency.md) | ||
| - [Graceful shutdown](guide/workers/graceful-shutdown.md) | ||
| - [Stalled Jobs](guide/workers/stalled-jobs.md) | ||
| - [Sandboxed processors](guide/workers/sandboxed-processors.md) | ||
| - [Pausing queues](guide/workers/pausing-queues.md) | ||
| - [Jobs](guide/jobs/README.md) | ||
| - [FIFO](guide/jobs/fifo.md) | ||
| - [LIFO](guide/jobs/lifo.md) | ||
| - [Job Ids](guide/jobs/job-ids.md) | ||
| - [Job Data](guide/jobs/job-data.md) | ||
| - [Debouncing](guide/jobs/debouncing.md) | ||
| - [Delayed](guide/jobs/delayed.md) | ||
| - [Repeatable](guide/jobs/repeatable.md) | ||
| - [Prioritized](guide/jobs/prioritized.md) | ||
| - [Removing jobs](guide/jobs/removing-job.md) | ||
| - [Stalled](guide/jobs/stalled.md) | ||
| - [Getters](guide/jobs/getters.md) | ||
| - [Job Schedulers](guide/job-schedulers/README.md) | ||
| - [Repeat Strategies](guide/job-schedulers/repeat-strategies.md) | ||
| - [Repeat options](guide/job-schedulers/repeat-options.md) | ||
| - [Manage Job Schedulers](guide/job-schedulers/manage-job-schedulers.md) | ||
| - [Flows](guide/flows/README.md) | ||
| - [Adding flows in bulk](guide/flows/adding-bulks.md) | ||
| - [Get Flow Tree](guide/flows/get-flow-tree.md) | ||
| - [Fail Parent](guide/flows/fail-parent.md) | ||
| - [Remove Dependency](guide/flows/remove-dependency.md) | ||
| - [Ignore Dependency](guide/flows/ignore-dependency.md) | ||
| - [Remove Child Dependency](guide/flows/remove-child-dependency.md) | ||
| - [Metrics](guide/metrics/metrics.md) | ||
| - [Rate limiting](guide/rate-limiting.md) | ||
| - [Parallelism and Concurrency](guide/parallelism-and-concurrency.md) | ||
| - [Retrying failing jobs](guide/retrying-failing-jobs.md) | ||
| - [Returning job data](guide/returning-job-data.md) | ||
| - [Events](guide/events.md) | ||
| - [QueueScheduler](guide/queuescheduler.md) | ||
| - [Redis™ Compatibility](guide/redis-tm-compatibility/README.md) | ||
| - [Dragonfly](guide/redis-tm-compatibility/dragonfly.md) | ||
| - [Redis™ hosting](guide/redis-tm-hosting/README.md) | ||
| - [AWS MemoryDB](guide/redis-tm-hosting/aws-memorydb.md) | ||
| - [AWS Elasticache](guide/redis-tm-hosting/aws-elasticache.md) | ||
| - [Architecture](guide/architecture.md) | ||
| - [NestJs](guide/nestjs/README.md) | ||
| - [Producers](guide/nestjs/producers.md) | ||
| - [Queue Events Listeners](guide/nestjs/queue-events-listeners.md) | ||
| - [Going to production](guide/going-to-production.md) | ||
| - [Migration to newer versions](guide/migration-to-newer-versions.md) | ||
| - [Troubleshooting](guide/troubleshooting.md) | ||
|
|
||
| ## Patterns | ||
|
|
||
| * [Adding jobs in bulk across different queues](patterns/adding-bulks.md) | ||
| * [Manually processing jobs](patterns/manually-fetching-jobs.md) | ||
| * [Named Processor](patterns/named-processor.md) | ||
| * [Flows](patterns/flows.md) | ||
| * [Idempotent jobs](patterns/idempotent-jobs.md) | ||
| * [Throttle jobs](patterns/throttle-jobs.md) | ||
| * [Process Step Jobs](patterns/process-step-jobs.md) | ||
| * [Failing fast when Redis is down](patterns/failing-fast-when-redis-is-down.md) | ||
| * [Stop retrying jobs](patterns/stop-retrying-jobs.md) | ||
| * [Timeout jobs](patterns/timeout-jobs.md) | ||
| * [Redis Cluster](patterns/redis-cluster.md) | ||
| - [Adding jobs in bulk across different queues](patterns/adding-bulks.md) | ||
| - [Manually processing jobs](patterns/manually-fetching-jobs.md) | ||
| - [Named Processor](patterns/named-processor.md) | ||
| - [Flows](patterns/flows.md) | ||
| - [Idempotent jobs](patterns/idempotent-jobs.md) | ||
| - [Throttle jobs](patterns/throttle-jobs.md) | ||
| - [Process Step Jobs](patterns/process-step-jobs.md) | ||
| - [Failing fast when Redis is down](patterns/failing-fast-when-redis-is-down.md) | ||
| - [Stop retrying jobs](patterns/stop-retrying-jobs.md) | ||
| - [Timeout jobs](patterns/timeout-jobs.md) | ||
| - [Redis Cluster](patterns/redis-cluster.md) | ||
|
|
||
| ## BullMQ Pro | ||
|
|
||
| * [Introduction](bullmq-pro/introduction.md) | ||
| * [Install](bullmq-pro/install.md) | ||
| * [Observables](bullmq-pro/observables/README.md) | ||
| * [Cancelation](bullmq-pro/observables/cancelation.md) | ||
| * [Groups](bullmq-pro/groups/README.md) | ||
| * [Getters](bullmq-pro/groups/getters.md) | ||
| * [Rate limiting](bullmq-pro/groups/rate-limiting.md) | ||
| * [Concurrency](bullmq-pro/groups/concurrency.md) | ||
| * [Local group concurrency](bullmq-pro/groups/local-group-concurrency.md) | ||
| * [Max group size](bullmq-pro/groups/max-group-size.md) | ||
| * [Pausing groups](bullmq-pro/groups/pausing-groups.md) | ||
| * [Prioritized intra-groups](bullmq-pro/groups/prioritized.md) | ||
| * [Sandboxes for groups](bullmq-pro/groups/sandboxes-for-groups.md) | ||
| * [Batches](bullmq-pro/batches.md) | ||
| * [NestJs](bullmq-pro/nestjs/README.md) | ||
| * [Producers](bullmq-pro/nestjs/producers.md) | ||
| * [Queue Events Listeners](bullmq-pro/nestjs/queue-events-listeners.md) | ||
| * [API Reference](https://nestjs.bullmq.pro/) | ||
| * [Changelog](bullmq-pro/nestjs/changelog.md) | ||
| * [API Reference](https://api.bullmq.pro) | ||
| * [Changelog](bullmq-pro/changelog.md) | ||
| * [Support](bullmq-pro/support.md) | ||
| - [Introduction](bullmq-pro/introduction.md) | ||
| - [Install](bullmq-pro/install.md) | ||
| - [Observables](bullmq-pro/observables/README.md) | ||
| - [Cancelation](bullmq-pro/observables/cancelation.md) | ||
| - [Groups](bullmq-pro/groups/README.md) | ||
| - [Getters](bullmq-pro/groups/getters.md) | ||
| - [Rate limiting](bullmq-pro/groups/rate-limiting.md) | ||
| - [Concurrency](bullmq-pro/groups/concurrency.md) | ||
| - [Local group concurrency](bullmq-pro/groups/local-group-concurrency.md) | ||
| - [Max group size](bullmq-pro/groups/max-group-size.md) | ||
| - [Pausing groups](bullmq-pro/groups/pausing-groups.md) | ||
| - [Prioritized intra-groups](bullmq-pro/groups/prioritized.md) | ||
| - [Sandboxes for groups](bullmq-pro/groups/sandboxes-for-groups.md) | ||
| - [Batches](bullmq-pro/batches.md) | ||
| - [NestJs](bullmq-pro/nestjs/README.md) | ||
| - [Producers](bullmq-pro/nestjs/producers.md) | ||
| - [Queue Events Listeners](bullmq-pro/nestjs/queue-events-listeners.md) | ||
| - [API Reference](https://nestjs.bullmq.pro/) | ||
| - [Changelog](bullmq-pro/nestjs/changelog.md) | ||
| - [API Reference](https://api.bullmq.pro) | ||
| - [Changelog](bullmq-pro/changelog.md) | ||
| - [Support](bullmq-pro/support.md) | ||
|
|
||
| ## Bull | ||
|
|
||
| * [Introduction](bull/introduction.md) | ||
| * [Install](bull/install.md) | ||
| * [Quick Guide](bull/quick-guide.md) | ||
| * [Important Notes](bull/important-notes.md) | ||
| * [Reference](https://github.com/OptimalBits/bull/blob/develop/REFERENCE.md) | ||
| * [Patterns](bull/patterns/README.md) | ||
| * [Persistent connections](bull/patterns/persistent-connections.md) | ||
| * [Message queue](bull/patterns/message-queue.md) | ||
| * [Returning Job Completions](bull/patterns/returning-job-completions.md) | ||
| * [Reusing Redis Connections](bull/patterns/reusing-redis-connections.md) | ||
| * [Redis cluster](bull/patterns/redis-cluster.md) | ||
| * [Custom backoff strategy](bull/patterns/custom-backoff-strategy.md) | ||
| * [Debugging](bull/patterns/debugging.md) | ||
| * [Manually fetching jobs](bull/patterns/manually-fetching-jobs.md) | ||
|
|
||
| ## Bull 3.x Migration | ||
|
|
||
| * [Compatibility class](bull-3.x-migration/compatibility-class.md) | ||
| * [Migration](bull-3.x-migration/migration.md) | ||
| - [Introduction](bull/introduction.md) | ||
| - [Install](bull/install.md) | ||
| - [Quick Guide](bull/quick-guide.md) | ||
| - [Important Notes](bull/important-notes.md) | ||
| - [Reference](https://github.com/OptimalBits/bull/blob/develop/REFERENCE.md) | ||
| - [Patterns](bull/patterns/README.md) | ||
| - [Persistent connections](bull/patterns/persistent-connections.md) | ||
| - [Message queue](bull/patterns/message-queue.md) | ||
| - [Returning Job Completions](bull/patterns/returning-job-completions.md) | ||
| - [Reusing Redis Connections](bull/patterns/reusing-redis-connections.md) | ||
| - [Redis cluster](bull/patterns/redis-cluster.md) | ||
| - [Custom backoff strategy](bull/patterns/custom-backoff-strategy.md) | ||
| - [Debugging](bull/patterns/debugging.md) | ||
| - [Manually fetching jobs](bull/patterns/manually-fetching-jobs.md) | ||
|
|
||
| ## Python | ||
|
|
||
| * [Introduction](python/introduction.md) | ||
| * [Changelog](python/changelog.md) | ||
| - [Introduction](python/introduction.md) | ||
| - [Changelog](python/changelog.md) |
This file was deleted.
Oops, something went wrong.
This file was deleted.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,53 @@ | ||
| --- | ||
| description: >- | ||
| Job Schedulers replace "repeatable jobs", and are available in v5.16.0 and | ||
| onwards | ||
| --- | ||
|
|
||
| # Job Schedulers | ||
|
|
||
| A Job Scheduler acts as a factory , producing jobs based on specified "repeat" settings. The Job Scheduler is highly flexible, accommodating various scenarios, including jobs produced at fixed intervals, according to cron expressions, or based on custom requirements. For historical reasons, jobs produced by the Job Scheduler are often referred to as ‘Repeatable Jobs’. | ||
|
|
||
| To create a scheduler, simply use the "upsertJobScheduler" method as demonstrated in the following example: | ||
|
|
||
| ```typescript | ||
| // Creates a new Job Scheduler that generates a job every 1000 milliseconds (1 second) | ||
| const firstJob = await queue.upsertJobScheduler('my-scheduler-id', { | ||
| every: 1000, | ||
| }); | ||
| ``` | ||
|
|
||
| This example will create a new Job Scheduler that will produce a new job every second. It will also return the first job created for this Job Scheduler, which will be in "delayed" status waiting to be processed after 1 second. | ||
|
|
||
| Now there are also a few important considerations that need to be explained here.: | ||
|
|
||
| - **Upsert vs. Add:** the 'upsert' is used instead of 'add' to simplify management of recurring jobs, especially in production deployments. It ensures the scheduler is updated or created without duplications. | ||
| - **Job Production Rate:** The scheduler will only generate new jobs when the last job begins processing. Therefore, if your queue is very busy, or if you do not have enough workers or concurrency, it is possible that you will get the jobs less frequently than the specified repetition interval. | ||
| - **Job Status:** As long as a Job Scheduler is producing jobs, there will be always one job associated to the scheduler in the "Delayed" status. | ||
|
|
||
| ### Using Job Templates | ||
|
|
||
| You can also define a template with standard names, data, and options for jobs added to a queue. This ensures that all jobs produced by the Job Scheduler inherit these settings: | ||
|
|
||
| ```typescript | ||
| // Create jobs every day at 3:15 (am) | ||
| const firstJob = await queue.upsertJobScheduler( | ||
| 'my-scheduler-id', | ||
| { pattern: '0 15 3 * * *' }, | ||
| { | ||
| name: 'my-job-name', | ||
| data: { foo: 'bar' }, | ||
| opts: { | ||
| backoff: 3, | ||
| attempts: 5, | ||
| removeOnFail: 1000, | ||
| }, | ||
| }, | ||
| ); | ||
| ``` | ||
|
|
||
| All jobs produced by this scheduler will use the given settings. Note that in the future you could call "upsertJobScheduler" again with the given "my-scheduler-id" in order to update any settings of this particular job scheduler, such as the repeat options or/and the job's template settings. | ||
|
|
||
| {% hint style="info" %} | ||
| Since jobs produced by the Job Scheduler will get a special job ID in order to guarantee that jobs will never be created more often than the given repeat settings, you cannot choose a custom job id. However you can use the job's name if you need to discriminate these jobs from other jobs. | ||
| {% endhint %} |
Oops, something went wrong.