Web Search

POST /api/v1/search returns ranked web pages, images, or news. Set type: "web" (the default) for titles, URLs, and snippets, type: "images" for images and their source pages, or type: "news" for news articles. Every accepted search creates a search run that saves the query, options, and ranked results. Set crawl: true to fetch the unique source pages within that same run. Search results and the run ID return immediately; page content arrives through the run results API. Use /api/v1/index/search to search content already indexed in your workspace.

Search the web

curl https://bulkgrid.com/api/v1/search \
  -H "X-API-Key: $BULKGRID_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"query":"open source web crawlers","limit":5,"country":"us","language":"en"}'

{
  "query": "open source web crawlers",
  "type": "web",
  "results": [
    {
      "title": "Example crawler",
      "url": "https://example.com/crawler",
      "snippet": "A crawler for collecting web content.",
      "position": 1
    }
  ],
  "count": 1,
  "run": { "id": "11111111-1111-4111-8111-111111111111", "status": "completed" }
}

query is required (1–500 characters). limit defaults to 10 and supports 1–10 results. country defaults to us, language to en; location can optionally narrow the search, for example Stockholm, Sweden. Results are ranked and may contain fewer entries than requested. Duplicate URLs and URLs without a valid HTTP(S) address are omitted.

Search images

curl https://bulkgrid.com/api/v1/search \
  -H "X-API-Key: $BULKGRID_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"query":"Stockholm skyline","type":"images","limit":5}'

{
  "query": "Stockholm skyline",
  "type": "images",
  "results": [
    {
      "title": "Stockholm waterfront",
      "url": "https://example.com/stockholm",
      "imageUrl": "https://example.com/stockholm.jpg",
      "imageWidth": 1920,
      "imageHeight": 1080,
      "thumbnailUrl": "https://example.com/stockholm-thumb.jpg",
      "position": 1
    }
  ],
  "count": 1,
  "run": { "id": "11111111-1111-4111-8111-111111111111", "status": "completed" }
}

url is the source page; imageUrl is the image itself. Dimensions are in pixels. imageWidth, imageHeight, and thumbnailUrl are null when unavailable. Image results are deduplicated by image URL, so multiple images from the same source page can appear. The response type identifies the result shape, including when results is empty. For image searches, crawl: true crawls each unique source page once. It does not download the image files.

Search news

curl https://bulkgrid.com/api/v1/search \
  -H "X-API-Key: $BULKGRID_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"query":"Stockholm transit","type":"news","limit":5,"country":"se"}'

{
  "query": "Stockholm transit",
  "type": "news",
  "results": [
    {
      "title": "Transit update",
      "url": "https://example.com/news",
      "snippet": "New city routes announced.",
      "source": "City News",
      "date": "2 hours ago",
      "imageUrl": "https://example.com/news.jpg",
      "position": 1
    }
  ],
  "count": 1,
  "run": { "id": "11111111-1111-4111-8111-111111111111", "status": "completed" }
}

source is the article publisher. date preserves the reported publication date text, which may be relative (for example, "2 hours ago") rather than an ISO timestamp. source, date, and imageUrl are null when unavailable. Articles are deduplicated by URL, and crawl: true fetches their pages in the same Search run.

Crawl the results

curl https://bulkgrid.com/api/v1/search \
  -H "X-API-Key: $BULKGRID_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"query":"open source web crawlers","limit":5,"crawl":true,"crawlOptions":{"formats":["markdown"]}}'

The top-level run always contains { "id": "<run-id>", "status": "pending" } when pages are queued, or completed for search-only and empty searches. Use GET /api/v1/runs/{runId} to reopen the saved query and results in its typed search field. Use GET /api/v1/runs/{runId}/results for crawled pages and their outcomes. Each page’s url matches a search result’s source page URL, so multiple images can share one page result. See Runs and results. In the dashboard, the Search run shows each result as a card with its title, URL, and search details. Crawl status updates automatically. Select View on a card to inspect its crawled content or error details as they become available. The crawler fetches each returned URL without following page links. Crawling is subject to normal concurrency and processed page limits. Empty searches save results: [] and complete without crawling. Search results remain available even if crawling fails; individual page failures appear alongside their results. If crawling cannot be queued, the response still contains the saved search results and a failed run; its last_error explains the failure. Retry a failed or cancelled run with POST /api/v1/runs/{runId}/retry. This creates a new Search run and reuses saved search results without another retrieval request. If the original search never produced results, retry performs and charges a new search. Retrying a Search run requires runs:retry and search:query. crawlOptions accepts normal crawl options except custom headers and includeDocumentLinks. These are rejected to prevent sending credentials to discovered sites or expanding the crawl beyond the search results. useLlmsTxt defaults to false so the crawler fetches the result pages themselves. crawlOptions requires crawl: true.

SDKs

const result = await client.search({
  query: 'open source web crawlers',
  limit: 5,
  crawl: true,
  crawlOptions: { formats: ['markdown'] },
});
console.log(result.results, result.run.id);
const run = await client.runs.get(result.run.id);
console.log(run.search); // Saved ranked results; run.status tracks optional crawling
const pages = await client.runs.results(result.run.id);

const images = await client.search({ query: 'Stockholm skyline', type: 'images' });
if (images.type === 'images') {
  console.log(images.results.map(image => image.imageUrl));
}

const news = await client.search({ query: 'Stockholm transit', type: 'news', crawl: true });
if (news.type === 'news') {
  console.log(news.results.map(article => ({ title: article.title, source: article.source, date: article.date })));
}

result = client.search({
    "query": "open source web crawlers",
    "limit": 5,
    "crawl": True,
    "crawlOptions": {"formats": ["markdown"]},
})
print(result["results"], result["run"]["id"])
run = client.runs.get(result["run"]["id"])
print(run.get("search"))
pages = client.runs.results(result["run"]["id"])

images = client.search({"query": "Stockholm skyline", "type": "images"})
if images["type"] == "images":
    print([image["imageUrl"] for image in images["results"]])

news = client.search({"query": "Stockholm transit", "type": "news", "crawl": True})
if news["type"] == "news":
    print([(article["title"], article["source"], article["date"]) for article in news["results"]])

Permissions and usage

Search requires search:query. Enabling crawling also requires runs:write; reading the run and its results requires runs:read and results:read respectively. Each accepted web search consumes one retrieval request before searching, including empty results and search service failures. Search service errors and timeouts return 502 with the failed run reference in details.run. Invalid input, missing configuration, and denied requests fail before creating a run. Bulkgrid does not automatically retry upstream search requests.

Get started

Index

More

Access and operations

Search the web

Search images

Search news

Crawl the results

SDKs

Permissions and usage

​Search the web

​Search images

​Search news

​Crawl the results

​SDKs

​Permissions and usage

Search the web

Search images

Search news

Crawl the results

SDKs

Permissions and usage