Bree

Bree is a Node.js and JavaScript job task scheduler with worker threads, cr...

QueueCRON
breejs/breejobscheduler.netbree

README

bree

build status code style styled with prettier made with lass license npm downloads
Bree is the best job scheduler for Node.js and JavaScript with cron, dates, ms, later, and human-friendly support.
Works in Node v12.17.0+, uses worker threads (Node.js) to spawn sandboxed processes, and supports async/await, retries, throttling, concurrency, and cancelable jobs with graceful shutdown. Simple, fast, and lightweight. Made for Forward Email and Lad.

Table of Contents


Foreword
Install
Upgrading
Usage and Examples
  ECMAScript modules (ESM)
  CommonJS (CJS)
Instance Options
Job Options
Job Interval and Timeout Values
Listening for events
Custom error/message handling
Cancellation, Retries, Stalled Jobs, and Graceful Reloading
Interval, Timeout, Date, and Cron Validation
Writing jobs with Promises and async-await
Callbacks, Done, and Completion States
Long-running jobs
Complex timeouts and intervals
Custom Worker Options
Using functions for jobs
Typescript and Usage with Bundlers
Concurrency
Plugins
  Available Plugins
  Creating plugins for Bree
Real-world usage
Contributors
License

Foreword


Bree was created to give you fine-grained control with simplicity, and has built-in support for workers, sandboxed processes, graceful reloading, cron jobs, dates, human-friendly time representations, and much more.

We recommend you to query a persistent database in your jobs, to prevent specific operations from running more than once.

Bree does not force you to use an additional database layer of [Redis][] or [MongoDB][] to manage job state.

In doing so, you should manage boolean job states yourself using queries.  For instance, if you have to send a welcome email to users, only send a welcome email to users that do not have a Date value set yet for welcome_email_sent_at.

Install


[npm][]:

  1. ```sh
  2. npm install bree
  3. ```

[yarn][]:

  1. ```sh
  2. yarn add bree
  3. ```

Upgrading


To see details about upgrading from the last major version, please see UPGRADING.md.

IMPORTANT: Bree v9.0.0 has several breaking changes, please see UPGRADING.md for more insight.


NOTE: Bree v6.5.0 is the last version to support Node v10 and browsers.


Usage and Examples


The example below assumes that you have a directory jobs in the root of the directory from which you run this example.  For example, if the example below is at /path/to/script.js, then /path/to/jobs/ must also exist as a directory.  If you wish to disable this feature, then pass root: false as an option.

Inside this jobs directory are individual scripts which are run using [Workers][] per optional timeouts, and additionally, an optional interval or cron expression.  The example below contains comments, which help to clarify how this works.

The option jobs passed to a new instance of Bree (as shown below) is an Array.  It contains values which can either be a String (name of a job in the jobs directory, which is run on boot) OR it can be an Object with name, path, timeout, and interval properties.  If you do not supply a path, then the path is created using the root directory (defaults to jobs) in combination with the name.  If you do not supply values for timeout and/nor interval, then these values are defaulted to 0 (which is the default for both, see index.js for more insight into configurable default options).

We have also documented all Instance Options and Job Options in this README below.  Be sure to read those sections so you have a complete understanding of how Bree works.

ECMAScript modules (ESM)


  1. ``` js
  2. // app.mjs

  4. import Bree from 'bree';

  6. const bree = new Bree({
  7.   // ... (see below) ...
  8. });

  10. // top-level await supported in Node v14.8+
  11. await bree.start();

  13. // ... (see below) ...
  14. ```

Please reference the #CommonJS example below for more insight and options.

CommonJS (CJS)


  1. ``` js
  2. // app.js

  4. const path = require('path');

  6. // optional
  7. const ms = require('ms');
  8. const dayjs = require('dayjs');
  9. const Graceful = require('@ladjs/graceful');
  10. const Cabin = require('cabin');

  12. // required
  13. const Bree = require('bree');

  15. //
  16. // NOTE: see the "Instance Options" section below in this README
  17. // for the complete list of options and their defaults
  18. //
  19. const bree = new Bree({
  20.   //
  21.   // NOTE: by default the `logger` is set to `console`
  22.   // however we recommend you to use CabinJS as it
  23.   // will automatically add application and worker metadata
  24.   // to your log output, and also masks sensitive data for you
  25.   //
  26.   //
  27.   // NOTE: You can also pass `false` as `logger: false` to disable logging
  28.   //
  29.   logger: new Cabin(),

  31.   //
  32.   // NOTE: instead of passing this Array as an option
  33.   // you can create a `./jobs/index.js` file, exporting
  34.   // this exact same array as `module.exports = [ ... ]`
  35.   // doing so will allow you to keep your job configuration and the jobs
  36.   // themselves all in the same folder and very organized
  37.   //
  38.   // See the "Job Options" section below in this README
  39.   // for the complete list of job options and configurations
  40.   //
  41.   jobs: [
  42.     // runs `./jobs/foo.js` on start
  43.     'foo',

  45.     // runs `./jobs/foo-bar.js` on start
  46.     {
  47.       name: 'foo-bar'
  48.     },

  50.     // runs `./jobs/some-other-path.js` on start
  51.     {
  52.       name: 'beep',
  53.       path: path.join(__dirname, 'jobs', 'some-other-path')
  54.     },

  56.     // runs `./jobs/worker-1.js` on the last day of the month
  57.     {
  58.       name: 'worker-1',
  59.       interval: 'on the last day of the month'
  60.     },

  62.     // runs `./jobs/worker-2.js` every other day
  63.     {
  64.       name: 'worker-2',
  65.       interval: 'every 2 days'
  66.     },

  68.     // runs `./jobs/worker-3.js` at 10:15am and 5:15pm every day except on Tuesday
  69.     {
  70.       name: 'worker-3',
  71.       interval: 'at 10:15 am also at 5:15pm except on Tuesday'
  72.     },

  74.     // runs `./jobs/worker-4.js` at 10:15am every weekday
  75.     {
  76.       name: 'worker-4',
  77.       cron: '15 10 ? * *',
  78.       cronValidate: {
  79.         override: {
  80.           useBlankDay: true
  81.         }
  82.       }
  83.     },

  85.     // runs `./jobs/worker-5.js` on after 10 minutes have elapsed
  86.     {
  87.       name: 'worker-5',
  88.       timeout: '10m'
  89.     },

  91.     // runs `./jobs/worker-6.js` after 1 minute and every 5 minutes thereafter
  92.     {
  93.       name: 'worker-6',
  94.       timeout: '1m',
  95.       interval: '5m'
  96.       // this is unnecessary but shows you can pass a Number (ms)
  97.       // interval: ms('5m')
  98.     },

  100.     // runs `./jobs/worker-7.js` after 3 days and 4 hours
  101.     {
  102.       name: 'worker-7',
  103.       // this example uses `human-interval` parsing
  104.       timeout: '3 days and 4 hours'
  105.     },

  107.     // runs `./jobs/worker-8.js` at midnight (once)
  108.     {
  109.       name: 'worker-8',
  110.       timeout: 'at 12:00 am'
  111.     },

  113.     // runs `./jobs/worker-9.js` every day at midnight
  114.     {
  115.       name: 'worker-9',
  116.       interval: 'at 12:00 am'
  117.     },

  119.     // runs `./jobs/worker-10.js` at midnight on the 1st of every month
  120.     {
  121.       name: 'worker-10',
  122.       cron: '0 0 1 * *'
  123.     },

  125.     // runs `./jobs/worker-11.js` at midnight on the last day of month
  126.     {
  127.       name: 'worker-11',
  128.       cron: '0 0 L * *',
  129.       cronValidate: {
  130.         useLastDayOfMonth: true
  131.       }
  132.     },

  134.     // runs `./jobs/worker-12.js` at a specific Date (e.g. in 3 days)
  135.     {
  136.       name: 'worker-12',
  137.       //
  138.       date: dayjs().add(3, 'days').toDate()
  139.       // you can also use momentjs
  140.       //
  141.       // date: moment('1/1/20', 'M/D/YY').toDate()
  142.       // you can pass Date instances (if it's in the past it will not get run)
  143.       // date: new Date()
  144.     },

  146.     // runs `./jobs/worker-13.js` on start and every 2 minutes
  147.     {
  148.       name: 'worker-13',
  149.       interval: '2m'
  150.     },

  152.     // runs `./jobs/worker-14.js` on start with custom `new Worker` options (see below)
  153.     {
  154.       name: 'worker-14',
  155.       //
  156.       worker: {
  157.         workerData: {
  158.           foo: 'bar',
  159.           beep: 'boop'
  160.         }
  161.       }
  162.     },

  164.     // runs `./jobs/worker-15.js` **NOT** on start, but every 2 minutes
  165.     {
  166.       name: 'worker-15',
  167.       timeout: false, // <-- specify `false` here to prevent default timeout (e.g. on start)
  168.       interval: '2m'
  169.     },

  171.     // runs `./jobs/worker-16.js` on January 1st, 2022
  172.     // and at midnight on the 1st of every month thereafter
  173.     {
  174.       name: 'worker-16',
  175.       date: dayjs('1-1-2022', 'M-D-YYYY').toDate(),
  176.       cron: '0 0 1 * *'
  177.     }
  178.   ]
  179. });

  181. // handle graceful reloads, pm2 support, and events like SIGHUP, SIGINT, etc.
  182. const graceful = new Graceful({ brees: [bree] });
  183. graceful.listen();

  185. // start all jobs (this is the equivalent of reloading a crontab):
  186. (async () => {
  187.   await bree.start();
  188. })();

  190. /*
  191. // start only a specific job:
  192. (async () => {
  193.   await bree.start('foo');
  194. })();

  196. // stop all jobs
  197. bree.stop();

  199. // stop only a specific job:
  200. bree.stop('beep');

  202. // run all jobs (this does not abide by timeout/interval/cron and spawns workers immediately)
  203. bree.run();

  205. // run a specific job (...)
  206. bree.run('beep');

  208. (async () => {
  209.   // add a job array after initialization:
  210.   const added = await bree.add(['boop']); // will return array of added jobs
  211.   // this must then be started using one of the above methods

  213.   // add a job after initialization:
  214.   await bree.add('boop');
  215.   // this must then be started using one of the above methods
  216. })();

  218. // remove a job after initialization:
  219. bree.remove('boop');
  220. */
  221. ```

For more examples - including setting up bree with TypeScript, ESModules, and implementing an Email Queue, see the examples folder.

For a more complete demo using express see: Bree Express Demo

Instance Options


Here is the full list of options and their defaults.  See src/index.js for more insight if necessary.

PropertyTypeDefaultDescription
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
`logger`Object`console`This
`root`String`path.resolve('jobs')`Resolves
`silenceRootCheckError`Boolean`false`Silences
`doRootCheck`Boolean`true`Attempts
`removeCompleted`Boolean`false`Removes
`timeout`Number`0`Default
`interval`Number`0`Default
`jobs`Array`[]`Defaults
`hasSeconds`Boolean`false`This
`cronValidate`Object`{}`This
`closeWorkerAfterMs`Number`0`If
`defaultRootIndex`String`index.js`This
`defaultExtension`String`js`This
`acceptedExtensions`Array`['.js',This
`worker`Object`{}`These
`outputWorkerMetadata`Boolean`false`By
`errorHandler`Function`null`Set
`workerMessageHandler`Function`null`Set

Job Options


See Interval, Timeout, Date, and Cron Validate below for more insight besides this table:

PropertyTypeDescription
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
`name`StringThe
`path`StringThe
`timeout`Number,Sets
`interval`Number,Sets
`date`DateThis
`cron`StringA
`hasSeconds`BooleanOverrides
`cronValidate`ObjectOverrides
`closeWorkerAfterMs`NumberOverrides
`worker`ObjectOverrides
`outputWorkerMetadata`BooleanOverrides

Job Interval and Timeout Values


These values can include Number, Object, and String variable types:

Number values indicates the number of milliseconds for the timeout or interval
Object values must be a [later][] schedule object value (e.g. `later.parse.cron('15 10 ? *'))`)
String values can be either a [later][], [human-interval][], or [ms][] String values (e.g. [later][] supports Strings such as every 5 mins, [human-interval][] supports Strings such as 3 days and 4 hours, and [ms][] supports Strings such as 4h for four hours)

Listening for events


Bree extends from EventEmitter and emits two events:

worker created with an argument of name
worker deleted with an argument of name

If you'd like to know when your workers are created (or deleted), you can do so through this example:

  1. ``` js
  2. bree.on('worker created', (name) => {
  3.   console.log('worker created', name);
  4.   console.log(bree.workers.get(name));
  5. });

  7. bree.on('worker deleted', (name) => {
  8.   console.log('worker deleted', name);
  9.   console.log(!bree.worker.has(name));
  10. });
  11. ```

Custom error/message handling


If you'd like to override default behavior for worker error/message handling, provide a callback function as errorHandler or workerMessageHandler parameter when creating a Bree instance.

NOTE: Any console.log calls, from within the worker, will not be sent to stdout/stderr until the main thread is available. Furthermore, any console.log calls, from within the worker, will not be sent if the process is terminated before the message is printed. You should use parentPort.postMessage() alongside errorHandler or workerMessageHandler to print to stdout/stderr during worker execution. This is a known bug for workers.


An example use-case. If you want to call an external service to record an error (like Honeybadger, Sentry, etc.) along with logging the error internally. You can do so with:

  1. ``` js
  2. const logger = ('../path/to/logger');
  3. const errorService = ('../path/to/error-service');

  5. new Bree({
  6.   jobs: [
  7.     {
  8.       name: 'job that sometimes throws errors',
  9.       path: jobFunction
  10.     }
  11.   ],
  12.   errorHandler: (error, workerMetadata) => {
  13.     // workerMetadata will be populated with extended worker information only if
  14.     // Bree instance is initialized with parameter `workerMetadata: true`
  15.     if (workerMetadata.threadId) {
  16.       logger.info(`There was an error while running a worker ${workerMetadata.name} with thread ID: ${workerMetadata.threadId}`)
  17.     } else {
  18.       logger.info(`There was an error while running a worker ${workerMetadata.name}`)
  19.     }

  21.     logger.error(error);
  22.     errorService.captureException(error);
  23.   }
  24. });
  25. ```

Cancellation, Retries, Stalled Jobs, and Graceful Reloading


We recommend that you listen for "cancel" event in your worker paths.  Doing so will allow you to handle graceful cancellation of jobs.  For example, you could use [p-cancelable][]

Here's a quick example of how to do that (e.g. ./jobs/some-worker.js):

  1. ``` js
  2. //
  3. const { parentPort } = require('worker_threads');

  5. // ...

  7. function cancel() {
  8.   // do cleanup here
  9.   // (if you're using @ladjs/graceful, the max time this can run by default is 5s)

  11.   // send a message to the parent that we're ready to terminate
  12.   // (you could do `process.exit(0)` or `process.exit(1)` instead if desired
  13.   // but this is a bit of a cleaner approach for worker termination
  14.   if (parentPort) parentPort.postMessage('cancelled');
  15.   else process.exit(0);
  16. }

  18. if (parentPort)
  19.   parentPort.once('message', message => {
  20.     if (message === 'cancel') return cancel();
  21.   });
  22. ```

If you'd like jobs to retry, simply wrap your usage of promises with [p-retry][].

We leave it up to you to have as much fine-grained control as you wish.

See [@ladjs/graceful][lad-graceful] for more insight into how this package works.

Interval, Timeout, Date, and Cron Validation


If you need help writing cron expressions, you can reference crontab.guru.

We support [later][], [human-interval][], or [ms][] String values for both timeout and interval.

If you pass a cron property, then it is validated against [cron-validate][].

You can pass a Date as the date property, but you cannot combine both date and timeout.

If you do pass a Date, then it is only run if it is in the future.

See Job Interval and Timeout Values above for more insight.

Writing jobs with Promises and async-await


If jobs are running with Node pre-v14.8.0, which enables top-level async-await support, here is the working alternative:

  1. ``` js
  2. const { parentPort } = require('worker_threads');

  4. const delay = require('delay');
  5. const ms = require('ms');

  7. (async () => {
  8.   // wait for a promise to finish
  9.   await delay(ms('10s'));

  11.   // signal to parent that the job is done
  12.   if (parentPort) parentPort.postMessage('done');
  13.   else process.exit(0);
  14. })();
  15. ```

Callbacks, Done, and Completion States


To close out the worker and signal that it is done, you can simply parentPort.postMessage('done'); and/or process.exit(0).

While writing your jobs (which will run in [worker][workers] threads), you should do one of the following:

Signal to the main thread that the process has completed by sending a "done" message (per the example above in Writing jobs with Promises and async-await)
Exit the process if there is NOT an error with code 0 (e.g. process.exit(0);)
Throw an error if an error occurs (this will bubble up to the worker event error listener and terminate it)
Exit the process if there IS an error with code 1 (e.g. process.exit(1))

Long-running jobs


If a job is already running, a new worker thread will not be spawned, instead logger.error will be invoked with an error message (no error will be thrown, don't worry).  This is to prevent bad practices from being used.  If you need something to be run more than one time, then make the job itself run the task multiple times.  This approach gives you more fine-grained control.

By default, workers run indefinitely and are not closed until they exit (e.g. via process.exit(0) or process.exit(1), OR send to the parent port a "close" message, which will subsequently call worker.close() to close the worker thread.

If you wish to specify a maximum time (in milliseconds) that a worker can run, then pass closeWorkerAfterMs (Number) either as a default option when creating a new Bree() instance (e.g. new Bree({ closeWorkerAfterMs: ms('10s') })) or on a per-job configuration, e.g. { name: 'beep', closeWorkerAfterMs: ms('5m') }.

As of v6.0.0 when you pass closeWorkerAfterMs, the timer will start once the worker is signaled as "online" (as opposed to previous versions which did not take this into account).

Complex timeouts and intervals


Since we use [later][], you can pass an instance of later.parse.recur, later.parse.cron, or later.parse.text as the timeout or interval property values (e.g. if you need to construct something manually).

You can also use [dayjs][] to construct dates (e.g. from now or a certain date) to millisecond differences using dayjs().diff(new Date(), 'milliseconds').  You would then pass that returned Number value as timeout or interval as needed.

Custom Worker Options


You can pass a default worker configuration object as new Bree({ worker: { ... } });.

These options are passed to the options argument when we internally invoke new Worker(path, options).

Additionally, you can pass custom worker options on a per-job basis through a worker property Object on the job definition.

See complete documentation for options (but you usually don't have to modify these).

Using functions for jobs


It is highly recommended to use files instead of functions. However, sometimes it is necessary to use functions.

You can pass a function to be run as a job:

  1. ``` js
  2. new Bree({ jobs: [someFunction] });
  3. ```

(or)

  1. ``` js
  2. new Bree({
  3.   jobs: [
  4.     {
  5.       name: 'job with function',
  6.       path: someFunction
  7.     }
  8.   ]
  9. });
  10. ```

The function will be run as if it's in its own file, therefore no variables or dependencies will be shared from the local context by default.

You should be able to pass data via worker.workerData (see Custom Worker Options).

Note that you cannot pass a built-in nor bound function.

Typescript and Usage with Bundlers


When working with a bundler or a tool that transpiles your code in some form or another, we recommend that your bundler is set up in a way that transforms both your application code and your jobs. Because your jobs are in their own files and are run in their own separate threads, they will not be part of your applications dependency graph and need to be setup as their own entry points. You need to ensure you have configured your tool to bundle your jobs into a jobs folder and keep them properly relative to your entry point folder.

We recommend setting the root instance options to path.join(__dirname,'jobs') so that bree searches for your jobs folder relative to the file being ran. (by default it searches for jobs relative to where node is invoked). We recommend treating each job as an entry point and running all jobs through the same transformations as your app code.

After an example transformation - you should expect the output in your dist folder to look like:

  1. ```tree
  2. - dist
  3.   |-jobs
  4.     |-job.js
  5.   |-index.js
  6. ```

For some example TypeScript set ups - see the examples folder.

For another alternative also see the @breejs/ts-worker plugin.

Concurrency


We recommend using the following packages in your workers for handling concurrency:

* * * *

Plugins


Plugins can be added to Bree using a similar method to Day.js

To add a plugin use the following method:

  1. ``` js
  2. Bree.extend(plugin, options);
  3. ```

Available Plugins


API
TypeScript Worker

Creating plugins for Bree


Plugins should be a function that recieves an options object and the Bree class:

  1. ``` js
  2.   const plugin = (options, Bree) => {
  3.     /* plugin logic */
  4.   };
  5. ```

Real-world usage


More detailed examples can be found in [Forward Email][forward-email], [Lad][], and [Ghost][ghost].

Contributors


NameWebsite
-------------------------------------------------
**Nick
**shadowgate15**

License


MIT © Nick Baugh

##

#

[ms]: https://github.com/vercel/ms

[human-interval]: https://github.com/agenda/human-interval

[npm]: https://www.npmjs.com/

[yarn]: https://yarnpkg.com/

[workers]: https://nodejs.org/api/worker_threads.html

[lad]: https://lad.js.org

[p-retry]: https://github.com/sindresorhus/p-retry

[p-cancelable]: https://github.com/sindresorhus/p-cancelable

[later]: https://breejs.github.io/later/parsers.html

[cron-validate]: https://github.com/Airfooox/cron-validate

[forward-email]: https://github.com/forwardemail/forwardemail.net

[dayjs]: https://github.com/iamkun/dayjs

[redis]: https://redis.io/

[mongodb]: https://www.mongodb.com/

[lad-graceful]: https://github.com/ladjs/graceful

[cabin]: https://cabinjs.com

[moment]: https://momentjs.com

[ghost]: https://ghost.org/