Piscina

A fast, efficient Node.js Worker Thread Pool implementation

Worker thread
piscinajs/piscinapiscina

README

piscina - the node.js worker pool


CI

✔ Fast communication between threads
✔ Covers both fixed-task and variable-task scenarios
✔ Supports flexible pool sizes
✔ Proper async tracking integration
✔ Tracking statistics for run and wait times
✔ Cancellation Support
✔ Supports enforcing memory resource limits
✔ Supports CommonJS, ESM, and TypeScript
✔ Custom task queues
✔ Optional CPU scheduling priorities on Linux

Written in TypeScript.

For Node.js 12.x and higher.

[MIT Licensed][].

Piscina API


Example


In main.js:

  1. ``` js
  2. const Piscina = require('piscina');

  4. const piscina = new Piscina({
  5.   filename: path.resolve(__dirname, 'worker.js')
  6. });

  8. (async function() {
  9.   const result = await piscina.runTask({ a: 4, b: 6 });
  10.   console.log(result);  // Prints 10
  11. })();
  12. ```

In worker.js:

  1. ``` js
  2. module.exports = ({ a, b }) => {
  3.   return a + b;
  4. };
  5. ```

The worker may also be an async function or may return a Promise:

  1. ``` js
  2. const { promisify } = require('util');
  3. const sleep = promisify(setTimeout);

  5. module.exports = async ({ a, b } => {
  6.   // Fake some async activity
  7.   await sleep(100);
  8.   return a + b;
  9. })
  10. ```

ESM is also supported for both Piscina and workers:

  1. ``` js
  2. import { Piscina } from 'piscina';

  4. const piscina = new Piscina({
  5.   // The URL must be a file:// URL
  6.   filename: new URL('./worker.mjs', import.meta.url).href
  7. });

  9. (async function () {
  10.   const result = await piscina.runTask({ a: 4, b: 6 });
  11.   console.log(result); // Prints 10
  12. })();
  13. ```

In worker.mjs:

  1. ``` js
  2. export default ({ a, b }) => {
  3.   return a + b;
  4. };
  5. ```

Cancelable Tasks


Submitted tasks may be canceled using either an AbortController or
an EventEmitter:

  1. ``` js
  2. 'use strict';

  4. const Piscina = require('piscina');
  5. const { AbortController } = require('abort-controller');
  6. const { resolve } = require('path');

  8. const piscina = new Piscina({
  9.   filename: resolve(__dirname, 'worker.js')
  10. });

  12. (async function() {
  13.   const abortController = new AbortController();
  14.   try {
  15.     const task = piscina.runTask({ a: 4, b: 6 }, abortController.signal);
  16.     abortController.abort();
  17.     await task;
  18.   } catch (err) {
  19.     console.log('The task was canceled');
  20.   }
  21. })();
  22. ```

To use AbortController, you will need to npm i abort-controller
(or yarn add abort-controller).

Alternatively, any EventEmitter that emits an 'abort' event
may be used as an abort controller:

  1. ``` js
  2. 'use strict';

  4. const Piscina = require('piscina');
  5. const EventEmitter = require('events');
  6. const { resolve } = require('path');

  8. const piscina = new Piscina({
  9.   filename: resolve(__dirname, 'worker.js')
  10. });

  12. (async function() {
  13.   const ee = new EventEmitter();
  14.   try {
  15.     const task = piscina.runTask({ a: 4, b: 6 }, ee);
  16.     ee.emit('abort');
  17.     await task;
  18.   } catch (err) {
  19.     console.log('The task was canceled');
  20.   }
  21. })();
  22. ```

Delaying Availability of Workers


A worker thread will not be made available to process tasks until Piscina
determines that it is "ready". By default, a worker is ready as soon as
Piscina loads it and acquires a reference to the exported handler function.

There may be times when the availability of a worker may need to be delayed
longer while the worker initializes any resources it may need to operate.
To support this case, the worker module may export a Promise that resolves
the handler function as opposed to exporting the function directly:

  1. ``` js
  2. async function initialize() {
  3.   await someAsyncInitializationActivity();
  4.   return ({ a, b }) => a + b;
  5. }

  7. module.exports = initialize();
  8. ```

Piscina will await the resolution of the exported Promise before marking
the worker thread available.

Backpressure


When the maxQueue option is set, once the Piscina queue is full, no
additional tasks may be submitted until the queue size falls below the
limit. The 'drain' event may be used to receive notification when the
queue is empty and all tasks have been submitted to workers for processing.

Example: Using a Node.js stream to feed a Piscina worker pool:
  1. ``` js
  2. 'use strict';

  4. const { resolve } = require('path');
  5. const Pool = require('../..');

  7. const pool = new Pool({
  8.   filename: resolve(__dirname, 'worker.js'),
  9.   maxQueue: 'auto'
  10. });

  12. const stream = getStreamSomehow();
  13. stream.setEncoding('utf8');

  15. pool.on('drain', () => {
  16.   if (stream.isPaused()) {
  17.     console.log('resuming...', counter, pool.queueSize);
  18.     stream.resume();
  19.   }
  20. });

  22. stream
  23.   .on('data', (data) => {
  24.     pool.runTask(data);
  25.     if (pool.queueSize === pool.options.maxQueue) {
  26.       console.log('pausing...', counter, pool.queueSize);
  27.       stream.pause();
  28.     }
  29.   })
  30.   .on('error', console.error)
  31.   .on('end', () => {
  32.     console.log('done');
  33.   });
  34. ```

Additional Examples


Additional examples can be found in the GitHub repo at
https://github.com/jasnell/piscina/tree/master/examples

Class: Piscina


Piscina works by creating a pool of Node.js Worker Threads to which
one or more tasks may be dispatched. Each worker thread executes a
single exported function defined in a separate file. Whenever a
task is dispatched to a worker, the worker invokes the exported
function and reports the return value back to Piscina when the
function completes.

This class extends [EventEmitter][] from Node.js.

Constructor: new Piscina([options])


The following optional configuration is supported:
  filename: (string | null) Provides the default source for the code that
    runs the tasks on Worker threads. This should be an absolute path or an
    absolute file:// URL to a file that exports a JavaScript function or
    async function as its default export or module.exports. [ES modules][]
    are supported.
  minThreads: (number) Sets the minimum number of threads that are always
    running for this thread pool. The default is based on the number of
    available CPUs.
  maxThreads: (number) Sets the maximum number of threads that are
    running for this thread pool. The default is based on the number of
    available CPUs.
  idleTimeout: (number) A timeout in milliseconds that specifies how long
    a Worker is allowed to be idle, i.e. not handling any tasks, before it is
    shut down. By default, this is immediate.
  maxQueue: (number | string) The maximum number of tasks that may be
    scheduled to run, but not yet running due to lack of available threads, at
    a given time. By default, there is no limit. The special value 'auto'
    may be used to have Piscina calculate the maximum as the square of maxThreads.
    When 'auto' is used, the calculated maxQueue value may be found by checking
    the [options.maxQueue](#property-options-readonly) property.
  concurrentTasksPerWorker: (number) Specifies how many tasks can share
    a single Worker thread simultaneously. The default is 1. This generally
    only makes sense to specify if there is some kind of asynchronous component
    to the task. Keep in mind that Worker threads are generally not built for
    handling I/O in parallel.
  useAtomics: (boolean) Use the [Atomics][] API for faster communication
    between threads. This is on by default.
  resourceLimits: (object) See [Node.js new Worker options][]
    maxOldGenerationSizeMb: (number) The maximum size of each worker threads
      main heap in MB.
    maxYoungGenerationSizeMb: (number) The maximum size of a heap space for
      recently created objects.
    codeRangeSizeMb: (number) The size of a pre-allocated memory range used
      for generated code.
  env: (object) If set, specifies the initial value of process.env inside
    the worker threads. See [Node.js new Worker options][] for details.
  argv: (any[]) List of arguments that will be stringified and appended to
    process.argv in the worker. See [Node.js new Worker options][] for details.
  execArgv: (string[]) List of Node.js CLI options passed to the worker.
    See [Node.js new Worker options][] for details.
  workerData: (any) Any JavaScript value that can be cloned and made
    available as require('piscina').workerData. See [Node.js new Worker options][]
    for details. Unlike regular Node.js Worker Threads, workerData must not
    specify any value requiring a transferList. This is because the workerData
    will be cloned for each pooled worker.
  taskQueue: (TaskQueue) By default, Piscina uses a first-in-first-out
    queue for submitted tasks. The taskQueue option can be used to provide an
    alternative implementation. See [Custom Task Queues][] for additional detail.
  niceIncrement: (number) An optional value that decreases priority for
    the individual threads, i.e. the higher the value, the lower the priority
    of the Worker threads. This value is only used on Linux and requires the
    optional [nice-napi][] module to be installed.
    See [nice(2)][] for more details.

Use caution when setting resource limits. Setting limits that are too low may
result in the Piscina worker threads being unusable.

Method: runTask(task[, transferList][, filename][, abortSignal])


Schedules a task to be run on a Worker thread.

task: Any value. This will be passed to the function that is exported from
  filename.
transferList: An optional lists of objects that is passed to
  [postMessage()] when posting task to the Worker, which are transferred
  rather than cloned.
filename: Optionally overrides the filename option passed to the
  constructor for this task. If no filename was specified to the constructor,
  this is mandatory.
abortSignal: An [AbortSignal][] instance. If passed, this can be used to
  cancel a task. If the task is already running, the corresponding Worker
  thread will be stopped.
  (More generally, any EventEmitter or EventTarget that emits 'abort'
  events can be passed here.) Abortable tasks cannot share threads regardless
  of the concurrentTasksPerWorker options.

This returns a Promise for the return value of the (async) function call
made to the function exported from filename. If the (async) function throws
an error, the returned Promise will be rejected with that error.
If the task is aborted, the returned Promise is rejected with an error
as well.

Method: destroy()


Stops all Workers and rejects all Promises for pending tasks.

This returns a Promise that is fulfilled once all threads have stopped.

Event: 'error'


An 'error' event is emitted by instances of this class when:

- Uncaught exceptions occur inside Worker threads that do not currently handle
  tasks.
- Unexpected messages are sent from from Worker threads.

All other errors are reported by rejecting the Promise returned from
runTask(), including rejections reported by the handler function itself.

Event: 'drain'


A 'drain' event is emitted whenever the queueSize reaches 0.

Property: completed (readonly)


The current number of completed tasks.

Property: duration (readonly)


The length of time (in milliseconds) since this Piscina instance was
created.

Property: options (readonly)


A copy of the options that are currently being used by this instance. This
object has the same properties as the options object passed to the constructor.

Property: runTime (readonly)


A histogram summary object summarizing the collected run times of completed
tasks. All values are expressed in milliseconds.

runTime.average {number} The average run time of all tasks
runTime.mean {number} The mean run time of all tasks
runTime.stddev {number} The standard deviation of collected run times
runTime.min {number} The fastest recorded run time
runTime.max {number} The slowest recorded run time

All properties following the pattern p{N} where N is a number (e.g. p1, p99)
represent the percentile distributions of run time observations. For example,
p99 is the 99th percentile indicating that 99% of the observed run times were
faster or equal to the given value.

  1. ``` js
  2. {
  3.   average: 1880.25,
  4.   mean: 1880.25,
  5.   stddev: 1.93,
  6.   min: 1877,
  7.   max: 1882.0190887451172,
  8.   p0_001: 1877,
  9.   p0_01: 1877,
  10.   p0_1: 1877,
  11.   p1: 1877,
  12.   p2_5: 1877,
  13.   p10: 1877,
  14.   p25: 1877,
  15.   p50: 1881,
  16.   p75: 1881,
  17.   p90: 1882,
  18.   p97_5: 1882,
  19.   p99: 1882,
  20.   p99_9: 1882,
  21.   p99_99: 1882,
  22.   p99_999: 1882
  23. }
  24. ```

Property: threads (readonly)


An Array of the Worker instances used by this pool.

Property: queueSize (readonly)


The current number of tasks waiting to be assigned to a Worker thread.

Property: utilization (readonly)


A point-in-time ratio comparing the approximate total mean run time
of completed tasks to the total runtime capacity of the pool.

A pools runtime capacity is determined by multiplying the duration
by the options.maxThread count. This provides an absolute theoretical
maximum aggregate compute time that the pool would be capable of.

The approximate total mean run time is determined by multiplying the
mean run time of all completed tasks by the total number of completed
tasks. This number represents the approximate amount of time the
pool as been actively processing tasks.

The utilization is then calculated by dividing the approximate total
mean run time by the capacity, yielding a fraction between 0 and 1.

Property: waitTime (readonly)


A histogram summary object summarizing the collected times tasks spent
waiting in the queue. All values are expressed in milliseconds.

waitTime.average {number} The average wait time of all tasks
waitTime.mean {number} The mean wait time of all tasks
waitTime.stddev {number} The standard deviation of collected wait times
waitTime.min {number} The fastest recorded wait time
waitTime.max {number} The longest recorded wait time

All properties following the pattern p{N} where N is a number (e.g. p1, p99)
represent the percentile distributions of wait time observations. For example,
p99 is the 99th percentile indicating that 99% of the observed wait times were
faster or equal to the given value.

  1. ``` js
  2. {
  3.   average: 1880.25,
  4.   mean: 1880.25,
  5.   stddev: 1.93,
  6.   min: 1877,
  7.   max: 1882.0190887451172,
  8.   p0_001: 1877,
  9.   p0_01: 1877,
  10.   p0_1: 1877,
  11.   p1: 1877,
  12.   p2_5: 1877,
  13.   p10: 1877,
  14.   p25: 1877,
  15.   p50: 1881,
  16.   p75: 1881,
  17.   p90: 1882,
  18.   p97_5: 1882,
  19.   p99: 1882,
  20.   p99_9: 1882,
  21.   p99_99: 1882,
  22.   p99_999: 1882
  23. }
  24. ```

Static property: isWorkerThread (readonly)


Is true if this code runs inside a Piscina threadpool as a Worker.

Static property: version (readonly)


Provides the current version of this library as a semver string.

Static method: move(value)


By default, any value returned by a worker function will be cloned when
returned back to the Piscina pool, even if that object is capable of
being transfered. The Piscina.move() method can be used to wrap and
mark transferable values such that they will by transfered rather than
cloned.

The value may be any object supported by Node.js to be transferable
(e.g. ArrayBuffer, any TypedArray, or MessagePort), or any object
implementing the Transferable interface.

  1. ``` js
  2. const { move } = require('piscina');

  4. module.exports = () => {
  5.   return move(new ArrayBuffer(10));
  6. }
  7. ```

The move() method will throw if the value is not transferable.

The object returned by the move() method should not be set as a
nested value in an object. If it is used, the move() object itself
will be cloned as opposed to transfering the object it wraps.

Interface: Transferable


Objects may implement the Transferable interface to create their own
custom transferable objects. This is useful when an object being
passed into or from a worker contains a deeply nested transferable
object such as an ArrayBuffer or MessagePort.

Transferable objects expose two properties inspected by Piscina
to determine how to transfer the object. These properties are
named using the special static Piscina.transferableSymbol and
Piscina.valueSymbol properties:

The Piscina.transferableSymbol property provides the object
  (or objects) that are to be included in the transferList.

The Piscina.valueSymbol property provides a surrogate value
  to transmit in place of the Transferable itself.

Both properties are required.

For example,

  1. ``` js
  2. const {
  3.   move,
  4.   transferableSymbol,
  5.   valueSymbol
  6. } = require('piscina');

  8. module.exports = () => {
  9.   const obj = {
  10.     a: { b: new Uint8Array(5); },
  11.     c: { new Uint8Array(10); },

  13.     get [transferableSymbol]() {
  14.       // Transfer the two underlying ArrayBuffers
  15.       return [this.a.b.buffer, this.c.buffer];
  16.     }

  18.     get [valueSymbol]() {
  19.       return { a: { b: this.b }, c: this.c };
  20.     }
  21.   };
  22.   return move(obj);
  23. };
  24. ```

Custom Task Queues


By default, Piscina uses a simple array-based first-in-first-out (fifo)
task queue. When a new task is submitted and there are no available
workers, tasks are pushed on to the queue until a worker becomes
available.

If the default fifo queue is not sufficient, user code may replace the
task queue implementation with a custom implementation using the
taskQueue option on the Piscina constructor.

Custom task queue objects must implement the TaskQueue interface,
described below using TypeScript syntax:

  1. ```ts
  2. interface Task {
  3.   readonly [Piscina.queueOptionsSymbol] : object | null;
  4. }

  6. interface TaskQueue {
  7.   readonly size : number;
  8.   shift () : Task | null;
  9.   remove (task : Task) : void;
  10.   push (task : Task) : void;
  11. }
  12. ```

An example of a custom task queue that uses a shuffled priority queue
is available in [examples/task-queue](./examples/task-queue/index.js);

The special symbol Piscina.queueOptionsSymbol may be set as a property
on tasks submitted to runTask() as a way of passing additional options
on to the custom TaskQueue implementation. (Note that because the
queue options are set as a property on the task, tasks with queue
options cannot be submitted as JavaScript primitives).

Current Limitations (Things we're working on / would love help with)


Improved Documentation
More examples
Benchmarks

Performance Notes


Workers are generally optimized for offloading synchronous,
compute-intensive operations off the main Node.js event loop thread.
While it is possible to perform asynchronous operations and I/O
within a Worker, the performance advantages of doing so will be
minimal.

Specifically, it is worth noting that asynchronous operations
within Node.js, including I/O such as file system operations
or CPU-bound tasks such as crypto operations or compression
algorithms, are already performed in parallel by Node.js and
libuv on a per-process level. This means that there will be
little performance impact on moving such async operations into
a Piscina worker (see examples/scrypt for example).

Queue Size


Piscina provides the ability to configure the minimum and
maximum number of worker threads active in the pool, as well as
set limits on the number of tasks that may be queued up waiting
for a free worker. It is important to note that setting the
maxQueue size too high relative to the number of worker threads
can have a detrimental impact on performance and memory usage.
Setting the maxQueue size too small can also be problematic
as doing so could cause your worker threads to become idle and
be shutdown. Our testing has shown that a maxQueue size of
approximately the square of the maximum number of threads is
generally sufficient and performs well for many cases, but this
will vary significantly depending on your workload. It will be
important to test and benchmark your worker pools to ensure you've
effectively balanced queue wait times, memory usage, and worker
pool utilization.

Queue Pressure and Idle Threads


The thread pool maintained by Piscina has both a minimum and maximum
limit to the number of threads that may be created. When a Piscina
instance is created, it will spawn the minimum number of threads
immediately, then create additional threads as needed up to the
limit set by maxThreads. Whenever a worker completes a task, a
check is made to determine if there is additional work for it to
perform. If there is no additional work, the thread is marked idle.
By default, idle threads are shutdown immediately, with Piscina
ensuring that the pool always maintains at least the minimum.

When a Piscina pool is processing a stream of tasks (for instance,
processing http server requests as in the React server-side
rendering example in examples/react-ssr), if the rate in which
new tasks are received and queued is not sufficient to keep workers
from going idle and terminating, the pool can experience a thrashing
effect -- excessively creating and terminating workers that will
cause a net performance loss. There are a couple of strategies to
avoid this churn:

Strategy 1: Ensure that the queue rate of new tasks is sufficient to
keep workers from going idle. We refer to this as "queue pressure".
If the queue pressure is too low, workers will go idle and terminate.
If the queue pressure is too high, tasks will stack up, experience
increased wait latency, and consume additional memory.

Strategy 2: Increase the idleTimeout configuration option. By
default, idle threads terminate immediately. The idleTimeout option
can be used to specify a longer period of time to wait for additional
tasks to be submitted before terminating the worker. If the queue
pressure is not maintained, this could result in workers sitting idle
but those will have less of a performance impact than the thrashing
that occurs when threads are repeatedly terminated and recreated.

Strategy 3: Increase the minThreads configuration option. This has
the same basic effect as increasing the idleTimeout. If the queue
pressure is not high enough, workers may sit idle indefinitely but
there will be less of a performance hit.

In applications using Piscina, it will be most effective to use a
combination of these three approaches and tune the various configuration
parameters to find the optimum combination both for the application
workload and the capabilities of the deployment environment. There
are no one set of options that are going to work best.

Thread priority on Linux systems


On Linux systems that support [nice(2)][], Piscina is capable of setting
the priority of every worker in the pool. To use this mechanism, an additional
optional native addon dependency (nice-napi, npm i nice-napi) is required.
Once [nice-napi][] is installed, creating a Piscina instance with the
niceIncrement configuration option will set the priority for the pool:

  1. ``` js
  2. const Piscina = require('piscina');
  3. const pool = new Piscina({
  4.   worker: '/absolute/path/to/worker.js',
  5.   niceIncrement: 20
  6. });
  7. ```

The higher the niceIncrement, the lower the CPU scheduling priority will be
for the pooled workers which will generally extend the execution time of
CPU-bound tasks but will help prevent those threads from stealing CPU time from
the main Node.js event loop thread. Whether this is a good thing or not depends
entirely on your application and will require careful profiling to get correct.

The key metrics to pay attention to when tuning the niceIncrement are the
sampled run times of the tasks in the worker pool (using the [runTime][]
property) and the [delay of the Node.js main thread event loop][].

Multiple Thread Pools and Embedding Piscina as a Dependency


Every Piscina instance creates a separate pool of threads and operates
without any awareness of the other. When multiple pools are created in a
single application the various threads may contend with one another, and
with the Node.js main event loop thread, and may cause an overall reduction
in system performance.

Modules that embed Piscina as a dependency should make it clear via
documentation that threads are being used. It would be ideal if those
would make it possible for users to provide an existing Piscina instance
as a configuration option in lieu of always creating their own.

Release Notes


1.6.1


Bug fix: Reject if AbortSignal is already aborted
Bug Fix: Use once listener for abort event

1.6.0


Add the niceIncrement configuration parameter.

1.5.1


Bug fixes around abortable task selection.

1.5.0


Added Piscina.move()
Added Custom Task Queues
Added utilization metric
Wait for workers to be ready before considering them as candidates
Additional examples

1.4.0


Added maxQueue = 'auto' to autocalculate the maximum queue size.
Added more examples, including an example of implementing a worker
  as a Node.js native addon.

1.3.0


Added the 'drain' event

1.2.0


Added support for ESM and file:// URLs
Added env, argv, execArgv, and workerData options
More examples

1.1.0


Added support for Worker Thread resourceLimits

1.0.0


Initial release!

The Team


* James M Snell * Anna Henningsen * Matteo Collina

Acknowledgements


Piscina development is sponsored by [NearForm Research][].

[Atomics]: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Atomics
[EventEmitter]: https://nodejs.org/api/events.html
[postMessage]: https://nodejs.org/api/worker_threads.html#worker_threads_port_postmessage_value_transferlist
[examples/task-queue]: https://github.com/jasnell/piscina/blob/master/examples/task-queue/index.js
[nice(2)]: https://linux.die.net/man/2/nice
[nice-napi]: https://npmjs.org/package/nice-napi
[runTime]: #property-runtime-readonly
[Custom Task Queues]: #custom_task_queues
[ES modules]: https://nodejs.org/api/esm.html
[Node.js new Worker options]: https://nodejs.org/api/worker_threads.html#worker_threads_new_worker_filename_options
[MIT Licensed]: LICENSE.md
[NearForm Research]: https://www.nearform.com/research/
[delay of the Node.js main thread event loop]: https://nodejs.org/dist/latest-v14.x/docs/api/perf_hooks.html#perf_hooks_perf_hooks_monitoreventloopdelay_options