Crawl

Turn webpages into content your AI can use.

Get Markdown, HTML, and links from one page, a list of URLs, or a website. Choose the scope, follow progress, and put the content to work in your application.

Try CrawlRead the docs
Single URLsBatch processingWebsite discoveryMarkdown, HTML, and links
From URL to content

Watch pages turn into usable content.

Follow a five-page example from discovery to processing. Get Markdown, HTML, and links, then try a real request in the playground.

Illustrative crawl

Example: five webpages are discovered and processed into Markdown, HTML, and links.

Sample data · no live requests

Try your own URL in the playground and preview the content returned by the API.

Try Crawl
Choose your scope

One page or a whole website

Start with the content your workflow needs. Use explicit URL lists when you know the pages, or discover pages as part of a deeper crawl.

Single URL

Get content from a documentation page, article, or product page. Use the Playground to try a URL, or submit it through the API or SDK.

Batch URLs

Process a known list of URLs in one run through the API or SDK. Retrieve each page’s result and inspect individual failures.

Website crawl

Start from a website and follow links within your chosen scope. Set page and depth limits, include or exclude paths, and restrict discovery to a site section.

Usable output

Ready for your next step

Use clean text for AI context, HTML for further processing, or a screenshot for a visual record.

Markdown, HTML, and links

Retrieve Markdown for text workflows, cleaned or original HTML for further processing, and links for discovery.

Browser rendering

Process JavaScript-driven pages with browser rendering. Configure a selector or extra wait time when content needs time to appear.

Screenshots when needed

Request a page or full-page screenshot alongside content when your workflow needs a visual record.

Build with the API

From request to usable content

Choose a single page, a batch, or a website crawl. Each example includes the steps to retrieve the result.

Set BULKGRID_API_KEY to a workspace API key in your backend environment. Required scopes: runs:write, runs:read, results:read.

Read the full guide · SDK examples

1. Start a crawl

Replace the example URL with your page. The response is a run object; save its id.

cURL
curl --fail-with-body "https://bulkgrid.com/api/v1/crawl" \
  -H "x-api-key: $BULKGRID_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
  "type": "crawl",
  "urls": [
    "https://example.com"
  ],
  "options": {
    "formats": [
      "markdown",
      "cleanHtml",
      "links"
    ]
  }
}'
2. Check progress

Set RUN_ID to the id returned above. Check every few seconds with a maximum wait time. Stop on completed, failed, or cancelled; inspect last_error on failure.

cURL
curl --fail-with-body "https://bulkgrid.com/api/v1/runs/$RUN_ID" \
  -H "x-api-key: $BULKGRID_API_KEY"
3. Retrieve results

After completion, list results. Check each error_message before using its output; paginate for larger runs.

cURL
curl --fail-with-body "https://bulkgrid.com/api/v1/runs/$RUN_ID/results?page=1&limit=20" \
  -H "x-api-key: $BULKGRID_API_KEY"
4. Read page content

Set RESULT_ID to the id of a successful result. This endpoint returns the requested Markdown as text.

cURL
curl --fail-with-body "https://bulkgrid.com/api/v1/runs/$RUN_ID/results/$RESULT_ID/content?type=markdown" \
  -H "x-api-key: $BULKGRID_API_KEY"
Build your workflow

Choose the next step

Use the capability that matches what you want to do with the web content.

Discover URLs with Map

Build a list from a website’s sitemaps before selecting pages to process.

Get fields with Extract

Submit URLs with a schema when your application needs structured JSON.

Maintain knowledge with Index

Create a source when you want ongoing refreshes and searchable passages.

Questions

Before you start

Choose the right scope and understand what you receive.

Put your first page to work

Open Crawl in the Playground, inspect the result, and expand to a batch or website when you are ready.

Try Crawl