website-scraper

Download website to local directory (including all css, images, js, etc.)

Scraping
website-scraper/node-website-scraperwebsite-scraper

README

Version Downloads Node.js CI Test Coverage Coverage Status


website-scraper


Options | Plugins | Log and debug | Frequently Asked Questions | Contributing | Code of Conduct


Download website to local directory (including all css, images, js, etc.)

Try it using demo app

Note: by default dynamic websites (where content is loaded by js) may be saved not correctly because website-scraper doesn't execute js, it only parses http responses for html and css files. If you need to download dynamic website take a look on website-scraper-puppeteer or website-scraper-phantom.

This module is an Open Source Software maintained by one developer in free time. If you want to thank the author of this module you can use GitHub Sponsors or Patreon.

Requirements

nodejs version >= 14.14
website-scraper v5 is pure ESM (it doesn't work with CommonJS), read more in release v5.0.0 docs

Installation

  1. ```
  2. npm install website-scraper
  3. ```

Usage

  1. ``` js
  2. import scrape from 'website-scraper'; // only as ESM, no CommonJS
  3. const options = {
  4.   urls: ['http://nodejs.org/'],
  5.   directory: '/path/to/save/'
  6. };

  8. // with async/await
  9. const result = await scrape(options);

  11. // with promise
  12. scrape(options).then((result) => {});
  13. ```

options

urls - urls to download,required
directory - path to save files,required
sources - selects which resources should be downloaded
recursive - follow hyperlinks in html files
maxRecursiveDepth - maximum depth for hyperlinks
maxDepth - maximum depth for all dependencies
request - custom options for http module got
subdirectories - subdirectories for file extensions
defaultFilename - filename for index page
prettifyUrls - prettify urls
ignoreErrors - whether to ignore errors on resource downloading
urlFilter - skip some urls
filenameGenerator - generate filename for downloaded resource
requestConcurrency - set maximum concurrent requests
plugins - plugins, allow to customize filenames, request options, response handling, saving to storage, etc.

Default options you can find in lib/config/defaults.js or get them using
  1. ``` js
  2. import defaultOptions from 'website-scraper/defaultOptions';
  3. ```

urls

Array of objects which contain urls to download and filenames for them. _Required_.
  1. ``` js
  2. scrape({
  3.   urls: [
  4.     'http://nodejs.org/', // Will be saved with default filename 'index.html'
  5.     {url: 'http://nodejs.org/about', filename: 'about.html'},
  6.     {url: 'http://blog.nodejs.org/', filename: 'blog.html'}
  7.   ],
  8.   directory: '/path/to/save'
  9. });
  10. ```

directory

String, absolute path to directory where downloaded files will be saved. Directory should not exist. It will be created by scraper. _Required_.
How to download website to existing directory and why it's not supported by default - check here.

sources

Array of objects to download, specifies selectors and attribute values to select files for downloading. By default scraper tries to download all possible resources. Scraper uses cheerio to select html elements so selector can be any selector that cheerio supports.
  1. ``` js
  2. // Downloading images, css files and scripts
  3. scrape({
  4.   urls: ['http://nodejs.org/'],
  5.   directory: '/path/to/save',
  6.   sources: [
  7.     {selector: 'img', attr: 'src'},
  8.     {selector: 'link[rel="stylesheet"]', attr: 'href'},
  9.     {selector: 'script', attr: 'src'}
  10.   ]
  11. });
  12. ```

recursive

Boolean, if true scraper will follow hyperlinks in html files. Don't forget to set maxRecursiveDepth to avoid infinite downloading. Defaults to false.

maxRecursiveDepth

Positive number, maximum allowed depth for hyperlinks. Other dependencies will be saved regardless of their depth. Defaults to null - no maximum recursive depth set.

maxDepth

Positive number, maximum allowed depth for all dependencies. Defaults to null - no maximum depth set.
In most of cases you need maxRecursiveDepth instead of this option.

The difference between maxRecursiveDepth and maxDepth is that
maxDepth is for all type of resources, so if you have
  > maxDepth=1 AND html (depth 0) ⟶ html (depth 1) ⟶ img (depth 2)

  last image will be filtered out by depth

maxRecursiveDepth is only for html resources, so if you have
  > maxRecursiveDepth=1 AND html (depth 0) ⟶ html (depth 1) ⟶ img (depth 2)

  only html resources with depth 2 will be filtered out, last image will be downloaded

request

Object, custom options for http module got which is used inside website-scraper. Allows to set retries, cookies, userAgent, encoding, etc.
  1. ``` js
  2. // use same request options for all resources
  3. scrape({
  4.   urls: ['http://example.com/'],
  5.   directory: '/path/to/save',
  6.   request: {
  7.     headers: {
  8.       'User-Agent': 'Mozilla/5.0 (Linux; Android 4.2.1; en-us; Nexus 4 Build/JOP40D) AppleWebKit/535.19 (KHTML, like Gecko) Chrome/18.0.1025.166 Mobile Safari/535.19'
  9.     }
  10.   }
  11. });
  12. ```

subdirectories

Array of objects, specifies subdirectories for file extensions. If null all files will be saved to directory.
  1. ``` js
  2. /* Separate files into directories:
  3.   - `img` for .jpg, .png, .svg (full path `/path/to/save/img`)
  4.   - `js` for .js (full path `/path/to/save/js`)
  5.   - `css` for .css (full path `/path/to/save/css`)
  6. */
  7. scrape({
  8.   urls: ['http://example.com'],
  9.   directory: '/path/to/save',
  10.   subdirectories: [
  11.     {directory: 'img', extensions: ['.jpg', '.png', '.svg']},
  12.     {directory: 'js', extensions: ['.js']},
  13.     {directory: 'css', extensions: ['.css']}
  14.   ]
  15. });
  16. ```

defaultFilename

String, filename for index page. Defaults to index.html.

prettifyUrls

Boolean, whether urls should be 'prettified', by having the defaultFilename removed. Defaults to false.

ignoreErrors

Boolean, if true scraper will continue downloading resources after error occurred, if false - scraper will finish process and return error. Defaults to false.

urlFilter

Function which is called for each url to check whether it should be scraped. Defaults to null - no url filter will be applied.
  1. ``` js
  2. // Links to other websites are filtered out by the urlFilter
  3. scrape({
  4.   urls: ['http://example.com/'],
  5.   urlFilter: function(url) {
  6.     return url.indexOf('http://example.com') === 0;
  7.   },
  8.   directory: '/path/to/save'
  9. });
  10. ```

filenameGenerator

String (name of the bundled filenameGenerator). Filename generator determines path in file system where the resource will be saved.

byType (default)
When the byType filenameGenerator is used the downloaded files are saved by extension (as defined by the subdirectories setting) or directly in the directory folder, if no subdirectory is specified for the specific extension.

bySiteStructure
When the bySiteStructure filenameGenerator is used the downloaded files are saved in directory using same structure as on the website:
- / => DIRECTORY/example.com/index.html
- /about => DIRECTORY/example.com/about/index.html
- //cdn.example.com/resources/jquery.min.js => DIRECTORY/cdn.example.com/resources/jquery.min.js

  1. ``` js
  2. scrape({
  3.   urls: ['http://example.com/'],
  4.   urlFilter: (url) => url.startsWith('http://example.com'), // Filter links to other websites
  5.   recursive: true,
  6.   maxRecursiveDepth: 10,
  7.   filenameGenerator: 'bySiteStructure',
  8.   directory: '/path/to/save'
  9. });
  10. ```

requestConcurrency

Number, maximum amount of concurrent requests. Defaults to Infinity.


plugins


Plugins allow to extend scraper behaviour

Built-in plugins
Existing plugins
Create plugin
Create action
    beforeStart
    afterFinish
    error
    beforeRequest
    afterResponse
    onResourceSaved
    onResourceError
    saveResource
    generateFilename
    getReference

Built-in plugins
Scraper has built-in plugins which are used by default if not overwritten with custom plugins. You can find them in lib/plugins directory. Thease plugins are intended for internal use but can be coppied if the behaviour of the plugins needs to be extended / changed.

Existing plugins
website-scraper-puppeteer - download dynamic (rendered with js) websites using puppeteer
website-scraper-phantom - download dynamic (rendered with js) websites using phantomJS
website-scraper-existing-directory - save files to existing directory

Create plugin

Plugin is object with .apply method, can be used to change scraper behavior.

.apply method takes one argument - registerAction function which allows to add handlers for different actions. Action handlers are functions that are called by scraper on different stages of downloading website. For example generateFilename is called to generate filename for resource based on its url, onResourceError is called when error occured during requesting/handling/saving resource.

You can add multiple plugins which register multiple actions. Plugins will be applied in order they were added to options.
All actions should be regular or async functions. Scraper will call actions of specific type in order they were added and use result (if supported by action type) from last action call.

List of supported actions with detailed descriptions and examples you can find below.
  1. ``` js
  2. class MyPlugin {
  3. apply(registerAction) {
  4.   registerAction('beforeStart', async ({options}) => {});
  5.   registerAction('afterFinish', async () => {});
  6.   registerAction('error', async ({error}) => {console.error(error)});
  7.   registerAction('beforeRequest', async ({resource, requestOptions}) => ({requestOptions}));
  8.   registerAction('afterResponse', async ({response}) => response.body);
  9.   registerAction('onResourceSaved', ({resource}) => {});
  10.   registerAction('onResourceError', ({resource, error}) => {});
  11.   registerAction('saveResource', async ({resource}) => {});
  12.   registerAction('generateFilename', async ({resource}) => {})
  13.   registerAction('getReference', async ({resource, parentResource, originalReference}) => {})
  14. }
  15. }

  17. scrape({
  18.   urls: ['http://example.com/'],
  19.   directory: '/path/to/save',
  20.   plugins: [ new MyPlugin() ]
  21. });
  22. ```

beforeStart
Action beforeStart is called before downloading is started. It can be used to initialize something needed for other actions.

Parameters - object which includes:
options - scraper normalized options object passed to scrape function
utils - scraper utils

  1. ``` js
  2. registerAction('beforeStart', async ({options, utils}) => {});
  3. ```

afterFinish
Action afterFinish is called after all resources downloaded or error occurred. Good place to shut down/close something initialized and used in other actions.

No parameters.
  1. ``` js
  2. registerAction('afterFinish', async () => {});
  3. ```

error
Action error is called when error occurred.

Parameters - object which includes:
error - Error object
  1. ``` js
  2. registerAction('error', async ({error}) => {console.log(error)});
  3. ```

beforeRequest
Action beforeRequest is called before requesting resource. You can use it to customize request options per resource, for example if you want to use different encodings for different resource types or add something to querystring.

Parameters - object which includes:
resource - Resource object
requestOptions - default options for http module got or options returned by previous beforeRequest action call

Should return object which includes custom options for got module.
If multiple actions beforeRequest added - scraper will use requestOptions from last one.
  1. ``` js
  2. // Add ?myParam=123 to querystring for resource with url 'http://example.com'
  3. registerAction('beforeRequest', async ({resource, requestOptions}) => {
  4. if (resource.getUrl() === 'http://example.com') {
  5.   return {requestOptions: extend(requestOptions, {searchParams: {myParam: 123}})};
  6. }
  7. return {requestOptions};
  8. });

  10. // Add random delays between requests
  11. registerAction('beforeRequest', async ({resource, requestOptions}) => {
  12. const time = Math.round(Math.random() * 10000);
  13. await new Promise((resolve) => setTimeout(resolve, time));
  14. return {requestOptions};
  15. });
  16. ```

afterResponse
Action afterResponse is called after each response, allows to customize resource or reject its saving.

Parameters - object which includes:
response - response object from http module got

Should return resolved Promise if resource should be saved or rejected with Error Promise if it should be skipped.
Promise should be resolved with:
the response object with the body modified in place as necessary.
or object with properties
  body (response body, string)
  encoding (binary or utf8) used to save the file, binary used by default.
  metadata (object) - everything you want to save for this resource (like headers, original text, timestamps, etc.), scraper will not use this field at all, it is only for result.
a binary string. This is advised against because of the binary assumption being made can foul up saving of utf8 responses to the filesystem.

If multiple actions afterResponse added - scraper will use result from last one.
  1. ``` js
  2. // Do not save resources which responded with 404 not found status code
  3. registerAction('afterResponse', ({response}) => {
  4. if (response.statusCode === 404) {
  5.    return null;
  6. } else {
  7.   // if you don't need metadata - you can just return Promise.resolve(response.body)
  8.   return {
  9.    body: response.body,
  10.    metadata: {
  11.     headers: response.headers,
  12.     someOtherData: [ 1, 2, 3 ]
  13.    },
  14.       encoding: 'utf8'
  15.   }
  16. }
  17. });
  18. ```

onResourceSaved
Action onResourceSaved is called each time after resource is saved (to file system or other storage with 'saveResource' action).

Parameters- object which includes:
resource - Resource object

Scraper ignores result returned from this action and does not wait until it is resolved
  1. ``` js
  2. registerAction('onResourceSaved', ({resource}) => console.log(`Resource ${resource.url} saved!`));
  3. ```

onResourceError
Action onResourceError is called each time when resource's downloading/handling/saving to was failed

Parameters- object which includes:
resource - Resource object
error - Error object

Scraper ignores result returned from this action and does not wait until it is resolved
  1. ``` js
  2. registerAction('onResourceError', ({resource, error}) => console.log(`Resource ${resource.url} has error ${error}`));
  3. ```

generateFilename
Action generateFilename is called to determine path in file system where the resource will be saved.

Parameters - object which includes:
resource - Resource object
responseData - object returned from afterResponse action, contains url, mimeType, body, metadata properties

Should return object which includes:
filename - String, relative to directory path for specified resource

If multiple actions generateFilename added - scraper will use result from last one.

Default plugins which generate filenames: byType, bySiteStructure
  1. ``` js
  2. // Generate random filename
  3. const crypto = require('crypto');
  4. registerAction('generateFilename', ({resource}) => {
  5.   return {filename: crypto.randomBytes(20).toString('hex')};
  6. });
  7. ```

getReference
Action getReference is called to retrieve reference to resource for parent resource. Can be used to customize reference to resource, for example, update missing resource (which was not loaded) with absolute url. By default reference is relative path from parentResource to resource (see GetRelativePathReferencePlugin).

Parameters - object which includes:
resource - Resource object
parentResource - Resource object
originalReference - string, original reference to resource in parentResource

Should return object which includes:
reference - string or null, reference to resource for parentResource. If you don't want to update reference - return null

If multiple actions getReference added - scraper will use result from last one.
  1. ``` js
  2. // Use relative filenames for saved resources and absolute urls for missing
  3. registerAction('getReference', ({resource, parentResource, originalReference}) => {
  4.   if (!resource) {
  5.     return {reference: parentResource.url + originalReference}
  6.   }
  7.   return {reference: utils.getRelativePath(parentResource.filename, resource.filename)};
  8. });
  9. ```


saveResource
Action saveResource is called to save file to some storage. Use it to save files where you need: to dropbox, amazon S3, existing directory, etc. By default all files are saved in local file system to new directory passed in directory option (see SaveResourceToFileSystemPlugin).

Parameters - object which includes:
resource - Resource object

If multiple actions saveResource added - resource will be saved to multiple storages.
  1. ``` js
  2. registerAction('saveResource', async ({resource}) => {
  3.   const filename = resource.getFilename();
  4.   const text = resource.getText();
  5.   await saveItSomewhere(filename, text);
  6. });
  7. ```

result

Array of Resource objects containing:
- url: url of loaded page
- filename: filename where page was saved (relative to directory)
- children: array of children Resources

Log and debug

This module uses debug to log events. To enable logs you should use environment variableDEBUG.
Next command will log everything from website-scraper
  1. ``` sh
  2. export DEBUG=website-scraper*; node app.js
  3. ```

Module has different loggers for levels: website-scraper:error, website-scraper:warn, website-scraper:info, website-scraper:debug, website-scraper:log. Please read debug documentation to find how to include/exclude specific loggers.