Running the spider

You can run the spider from your Node.js environment of choice (e.g. AWS Lambda, Google Cloud Function, local Node.js script, etc). Once you have made sure your Node environment has the project peer dependencies installed (see Installation section), in your program entry point or any JS/TS module of choice, import the crawlSite method from the anansi-js package, then invoke it with the required arguments:

import { crawlSite } from 'anansi-js';

const letsStartCrawling = async () => {
  await crawlSite({
      configFilePath: 'path/to/your/config.json',
      searchEngineOpts: {
        algolia: {
          apiKey: '<your algolia API key>',
          appId: '<your algolia app ID>',
          indexName: '<your algolia index name>'
        }
      }
    });
}

letsStartCrawling().then(() => {
  process.exit(0);
})