> ## Documentation Index
> Fetch the complete documentation index at: https://bulkgrid.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Web Search

> Search web pages, images, or news and optionally crawl their source pages.

`POST /api/v1/search` returns ranked web pages, images, or news. Set `type: "web"` (the default) for titles, URLs, and snippets, `type: "images"` for images and their source pages, or `type: "news"` for news articles. Every accepted search creates a `search` run that saves the query, options, and ranked results. Set `crawl: true` to fetch the unique source pages within that same run. Search results and the run ID return immediately; page content arrives through the run results API.

Use `/api/v1/index/search` to search content already indexed in your workspace.

## Search the web

```bash theme={null}
curl https://bulkgrid.com/api/v1/search \
  -H "X-API-Key: $BULKGRID_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"query":"open source web crawlers","limit":5,"country":"us","language":"en"}'
```

```json theme={null}
{
  "query": "open source web crawlers",
  "type": "web",
  "results": [
    {
      "title": "Example crawler",
      "url": "https://example.com/crawler",
      "snippet": "A crawler for collecting web content.",
      "position": 1
    }
  ],
  "count": 1,
  "run": { "id": "11111111-1111-4111-8111-111111111111", "status": "completed" }
}
```

`query` is required (1–500 characters). `limit` defaults to 10 and supports 1–10 results. `country` defaults to `us`, `language` to `en`; `location` can optionally narrow the search, for example `Stockholm, Sweden`. Results are ranked and may contain fewer entries than requested. Duplicate URLs and URLs without a valid HTTP(S) address are omitted.

## Search images

```bash theme={null}
curl https://bulkgrid.com/api/v1/search \
  -H "X-API-Key: $BULKGRID_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"query":"Stockholm skyline","type":"images","limit":5}'
```

```json theme={null}
{
  "query": "Stockholm skyline",
  "type": "images",
  "results": [
    {
      "title": "Stockholm waterfront",
      "url": "https://example.com/stockholm",
      "imageUrl": "https://example.com/stockholm.jpg",
      "imageWidth": 1920,
      "imageHeight": 1080,
      "thumbnailUrl": "https://example.com/stockholm-thumb.jpg",
      "position": 1
    }
  ],
  "count": 1,
  "run": { "id": "11111111-1111-4111-8111-111111111111", "status": "completed" }
}
```

`url` is the source page; `imageUrl` is the image itself. Dimensions are in pixels. `imageWidth`, `imageHeight`, and `thumbnailUrl` are `null` when unavailable. Image results are deduplicated by image URL, so multiple images from the same source page can appear. The response `type` identifies the result shape, including when `results` is empty.

For image searches, `crawl: true` crawls each unique source page once. It does not download the image files.

## Search news

```bash theme={null}
curl https://bulkgrid.com/api/v1/search \
  -H "X-API-Key: $BULKGRID_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"query":"Stockholm transit","type":"news","limit":5,"country":"se"}'
```

```json theme={null}
{
  "query": "Stockholm transit",
  "type": "news",
  "results": [
    {
      "title": "Transit update",
      "url": "https://example.com/news",
      "snippet": "New city routes announced.",
      "source": "City News",
      "date": "2 hours ago",
      "imageUrl": "https://example.com/news.jpg",
      "position": 1
    }
  ],
  "count": 1,
  "run": { "id": "11111111-1111-4111-8111-111111111111", "status": "completed" }
}
```

`source` is the article publisher. `date` preserves the reported publication date text, which may be relative (for example, `"2 hours ago"`) rather than an ISO timestamp. `source`, `date`, and `imageUrl` are `null` when unavailable. Articles are deduplicated by URL, and `crawl: true` fetches their pages in the same Search run.

## Crawl the results

```bash theme={null}
curl https://bulkgrid.com/api/v1/search \
  -H "X-API-Key: $BULKGRID_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"query":"open source web crawlers","limit":5,"crawl":true,"crawlOptions":{"formats":["markdown"]}}'
```

The top-level `run` always contains `{ "id": "<run-id>", "status": "pending" }` when pages are queued, or `completed` for search-only and empty searches. Use `GET /api/v1/runs/{runId}` to reopen the saved query and results in its typed `search` field. Use `GET /api/v1/runs/{runId}/results` for crawled pages and their outcomes. Each page's `url` matches a search result's source page URL, so multiple images can share one page result. See [Runs and results](/docs/runs-and-results).

In the dashboard, the Search run shows each result as a card with its title, URL, and search details. Crawl status updates automatically. Select **View** on a card to inspect its crawled content or error details as they become available.

The crawler fetches each returned URL without following page links. Crawling is subject to normal concurrency and processed page limits. Empty searches save `results: []` and complete without crawling. Search results remain available even if crawling fails; individual page failures appear alongside their results. If crawling cannot be queued, the response still contains the saved search results and a failed `run`; its `last_error` explains the failure.

Retry a failed or cancelled run with `POST /api/v1/runs/{runId}/retry`. This creates a new Search run and reuses saved search results without another retrieval request. If the original search never produced results, retry performs and charges a new search. Retrying a Search run requires `runs:retry` and `search:query`.

`crawlOptions` accepts normal crawl options except custom `headers` and `includeDocumentLinks`. These are rejected to prevent sending credentials to discovered sites or expanding the crawl beyond the search results. `useLlmsTxt` defaults to `false` so the crawler fetches the result pages themselves. `crawlOptions` requires `crawl: true`.

## SDKs

```typescript theme={null}
const result = await client.search({
  query: 'open source web crawlers',
  limit: 5,
  crawl: true,
  crawlOptions: { formats: ['markdown'] },
});
console.log(result.results, result.run.id);
const run = await client.runs.get(result.run.id);
console.log(run.search); // Saved ranked results; run.status tracks optional crawling
const pages = await client.runs.results(result.run.id);

const images = await client.search({ query: 'Stockholm skyline', type: 'images' });
if (images.type === 'images') {
  console.log(images.results.map(image => image.imageUrl));
}

const news = await client.search({ query: 'Stockholm transit', type: 'news', crawl: true });
if (news.type === 'news') {
  console.log(news.results.map(article => ({ title: article.title, source: article.source, date: article.date })));
}
```

```python theme={null}
result = client.search({
    "query": "open source web crawlers",
    "limit": 5,
    "crawl": True,
    "crawlOptions": {"formats": ["markdown"]},
})
print(result["results"], result["run"]["id"])
run = client.runs.get(result["run"]["id"])
print(run.get("search"))
pages = client.runs.results(result["run"]["id"])

images = client.search({"query": "Stockholm skyline", "type": "images"})
if images["type"] == "images":
    print([image["imageUrl"] for image in images["results"]])

news = client.search({"query": "Stockholm transit", "type": "news", "crawl": True})
if news["type"] == "news":
    print([(article["title"], article["source"], article["date"]) for article in news["results"]])
```

## Permissions and usage

Search requires `search:query`. Enabling crawling also requires `runs:write`; reading the run and its results requires `runs:read` and `results:read` respectively.

Each accepted web search consumes one retrieval request before searching, including empty results and search service failures. Search service errors and timeouts return `502` with the failed run reference in `details.run`. Invalid input, missing configuration, and denied requests fail before creating a run. Bulkgrid does not automatically retry upstream search requests.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.