📄️ Installation
Install the project's npm package and peer dependencies (puppeteer and puppeteer-cluster)
📄️ Creating a config.json file
In the repo where you will be running the spider from, create a new config.json file.
📄️ Choosing a search plugin
The spider can work with a Search Plugin to index scraped page records in a 3rd party search engine. Currently only Algolia is supported.
📄️ Running the spider
You can run the spider from your Node.js environment of choice (e.g. AWS Lambda, Google Cloud Function, local Node.js script, etc).