Find the web content your AI needs.
Search webpages, images, or news with one API. Get ranked results with source URLs, then optionally crawl the result pages for Markdown and HTML.
Find the pages. Get the content.
Start with a query and ranked results. Enable crawling when your application needs the content behind the links.
Search · query to context
Illustrative resultsopen source web crawlers
Ranked results with source URLs
- 1
docs.example.com/crawler
Crawler documentation
Installation, configuration, and your first crawl.
- 2
example.com/guides/crawling
Compare crawling approaches
Choose how to collect the pages your application needs.
- 3
example.com/tutorials/content
Build a content pipeline
From discovered URLs to useful page content.
Go beyond the snippet.
Enable crawling to collect the result pages. Retrieve Markdown and HTML when processing finishes.
Example Markdown · after crawl
# Crawler documentation
## Getting started
Install the crawler, choose your URLs, and collect the content your application needs.Search results arrive first. Page content follows in the same run.
Sample data. This preview does not run a search. Try your own query in the workspace playground.
Try SearchOne API. Three ways to explore.
Choose web, images, or news for each request. Every result includes a source page URL you can use in your next step.
Webpages
Get ranked titles, URLs, snippets, and positions. Find documentation, company pages, articles, and other public web content.
Images
Find image URLs and their source pages, with dimensions and thumbnails when available. Keep a reference to where each image came from.
News
Find articles with titles, links, and snippets. Publisher names, reported publication dates, and images are included when available.
Search for your audience and workflow.
Keep requests focused with the controls your application needs, then decide how much content to collect.
Set the location and language
Choose a country and language, and optionally specify a location such as a city. Localize your query for the audience you serve.
Choose how many results you need
Request up to 10 results. Bulkgrid returns ranked, deduplicated results and their source URLs so your application can decide what to use.
Collect the pages behind the results
Enable crawling to fetch each unique source page without following its links. Read Markdown or HTML once processing finishes, and inspect individual page outcomes.
Start with a query. Build from the results.
Try web, image, or news search. Add crawling when you need page content, then follow the run to retrieve it.
Set BULKGRID_API_KEY to a workspace API key in your backend environment. Required scopes: search:query; add runs:write for crawling, runs:read and results:read to follow the run and read page content.
Read the full guide · SDK examples
1. Search the web
Replace the query. Read ranked titles, URLs, snippets, and positions in results. The response also includes run.id and run.status for the saved Search run.
curl --fail-with-body "https://bulkgrid.com/api/v1/search" \
-H "x-api-key: $BULKGRID_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"query": "open source web crawlers",
"type": "web",
"limit": 5,
"country": "us",
"language": "en"
}'Connect discovery to useful knowledge.
Search finds results across the public web. Use the other capabilities when you need a different scope or output.
Explore a known website with Map
Already know the website? Discover URLs from its sitemaps and select the pages to process.
Turn selected pages into data with Extract
Use URLs from your results in a separate extraction request. Apply your schema to get the fields your application needs.
Maintain selected sources with Index
Connect the public and private sources you want to keep. Index manages storage, scheduled refreshes, and retrieval from that knowledge.
Before you search
Understand the result types, page content, and how Search fits your workflow.
Give your next workflow a view of the web.
Open the workspace playground, enter a query, and inspect the results. Enable crawling when you need the page content too.
Try Search