Map

Find the right pages before you crawl.

Turn a website’s sitemaps into a focused list of URLs. Filter for the pages that matter, choose your scope, and use the list in your next crawl or extraction.

Try MapRead the docs
Sitemap discoveryURL keyword filtersSubdomain controlsCopy or download URLs
See the output

See which pages are worth collecting.

Explore a real result in the shared playground. One free preview; create an account to unlock the full output.

A website URL

Discover the pages you need

Try it in the playground

Example output

/docs/getting-started
/docs/api
/docs/integrations
Discover pages

Spend your crawl on the pages that matter

Find the sitemap URLs you need before fetching page content. Focus on product pages, documentation, or another section of the site.

Discover from sitemaps

Find URLs listed in the site’s sitemaps, including sitemap indexes that point to additional sitemap files.

Filter by URL keywords

Provide search words to narrow the list by matches in URL text. Use terms such as docs, pricing, or integrations to focus your selection.

Set clear limits

Choose whether to include subdomains, remove query parameters before deduplication, and cap the number of returned URLs.

Build with the API

Discover URLs with one request

Start a map run, then read its URL list and any discovery warnings. Pass the pages you choose to your next request.

Set BULKGRID_API_KEY to a workspace API key in your backend environment. Required scopes: runs:write, runs:read.

Read the full guide · SDK examples

1. Map a website

Replace the website URL. Filter sitemap URLs by keywords and save the returned run id.

cURL
curl --fail-with-body "https://bulkgrid.com/api/v1/map" \
  -H "x-api-key: $BULKGRID_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
  "type": "map",
  "url": "https://example.com",
  "config": {
    "search": "products",
    "limit": 100,
    "includeSubdomains": false
  }
}'
2. Check progress

Set RUN_ID to the returned id. Poll with a maximum wait time until completed, failed, or cancelled. On completion, read result_data.links and result_data.warnings from this response. Submit chosen URLs to Crawl or Extract separately.

cURL
curl --fail-with-body "https://bulkgrid.com/api/v1/runs/$RUN_ID" \
  -H "x-api-key: $BULKGRID_API_KEY"
Put the list to work

Process the pages you choose

Use the URL list as input to the workflow you need. You decide which pages to process.

Crawl selected URLs

Retrieve Markdown, HTML, or links from the pages you select.

Extract specific fields

Submit the selected URLs with the schema your workflow needs.

Create a focused index

Use selected pages to define a source for ongoing refresh and retrieval.

Questions

A few things to know

The details that help you choose and build.

Choose the pages worth processing

Map a website in the Playground, review its sitemap URLs, and build the list your workflow needs.

Try Map