Deep Crawl

​

Best fit

• you do not want to maintain a full URL list manually
• you need broader documentation or help-center coverage
• you want path controls around a site section

​

Request shape

curl "$BULKGRID_BASE_URL/api/v1/deep-crawl" \
  -H 'Content-Type: application/json' \
  -H "x-api-key: $BULKGRID_API_KEY" \
  -d '{
    "url": "https://example.com/docs",
    "config": {
      "maxDepth": 3,
      "maxPages": 100,
      "includePaths": ["/docs", "/blog"],
      "excludePaths": ["/legal"],
      "includeExternal": false,
      "includeDocumentLinks": true,
      "restrictToStartPath": true
    },
    "options": {
      "formats": ["markdown", "cleanHtml", "links"],
      "timeout": 30000,
      "blockAds": true,
      "useInteractions": true
    }
  }'

​

Key controls

• maxDepth: how far Bulkgrid should traverse from the starting URL
• maxPages: upper bound on discovered pages to process
• includePaths: preferred allowed path prefixes
• excludePaths: path areas to avoid
• includeExternal: whether external domains may be followed
• includeDocumentLinks: whether document links should be included
• restrictToStartPath: whether the crawl should stay under the starting path

​

Recommendation