1
Webhook Delivery
If a delivery fails, we retry with exponential backoff for up to 24 hours. Return a 2xx response to acknowledge the event and stop further attempts.
https://example.com/docs/webhooks
Index
Make websites and docs searchable for AI agents
Connect websites and GitHub documentation. Bulkgrid indexes the content, keeps it refreshed, and returns relevant passages with links to the original sources.
Example animation: index three documents from example.com, then search for webhook retry behavior. Retrieve two relevant passages with links to their original pages.
Benefits
Better Answers Start with Better Context
Retrieve focused evidence, keep it current, and let Bulkgrid manage the indexing infrastructure.
Spend Fewer Tokens
Retrieve the right source context instead of sending large pages, pasted docs, or noisy search results into every prompt.
Improve Answer Accuracy
Ground responses in current, source-backed evidence to reduce hallucinations and avoid relying on stale model memory.
Use Fresh Source Data
Keep important websites automatically recrawled and updated so AI is not working from old content.
Query Instantly
Search already-indexed source content immediately instead of crawling pages at question time.
No Infrastructure Maintenance
Bulkgrid handles crawling, rendering, retries, storage, indexing, recrawls, and change detection.
Change Detection
See when pages are added, removed, updated, or fail so teams can react without checking manually.
Outputs
Find the Context Your AI Needs.
Give your application the context behind an answer. Retrieve focused passages from your indexed content, with source URLs and document metadata in every result.
- Find passages using both exact terms and meaning.
- Keep the source URL alongside the retrieved content.
- Pass structured results to your application or AI agent.
How are failed webhooks retried?
2 passages · Product Documentation
1
Webhook Delivery
If a delivery fails, we retry with exponential backoff for up to 24 hours. Return a 2xx response to acknowledge the event and stop further attempts.
https://example.com/docs/webhooks
2
Retry Policy
Each delivery attempt includes the same event ID. Store this ID to handle duplicate deliveries safely. You can replay a failed event from the delivery log.
https://example.com/api/retries
Collections
Curate Knowledge for Your AI Agent
Bring the right sources together in a collection. Include the folders and pages that matter, and leave unrelated content out of your search.
- Combine public and private sources you can access.
- Include or exclude whole sources, folders, or individual URLs.
- Limit API keys and OAuth connections to approved collections.
Collection
Product Documentation
example.com
How are failed webhooks retried?
2 passages in this collection preview
Webhook Delivery
https://example.com/docs/webhooks
Retry Policy
https://example.com/api/retries
Features
From Sources to Searchable Context
Keep your knowledge organized and current while Bulkgrid handles indexing, storage, and retrieval.
Hybrid Search
Combine keyword and semantic search to match exact names, technical terms, and the meaning behind a question.
Websites and GitHub
Index website content or sync Markdown and MDX from GitHub. Choose website paths or repository folders to include.
Scheduled Refreshes
Keep each source updated on its configured schedule, or request a refresh. New content becomes searchable after indexing completes.
Source References
Get passages with source URLs and document metadata, so your application can cite its context and readers can check the original.
Public and Private Sources
Reuse shared public indexes or maintain private sources in your workspace. Collections and scoped credentials control what an application can search.
Visible Sync History
Follow indexing progress, inspect failures, and see the pages and files added, changed, or removed in each run.
SDKs
Search Your Knowledge with a Few Lines of Code
Use the TypeScript or Python SDK to retrieve passages from an indexed collection. Add sources and organize them into collections through the same client.
search-collection.ts
import { BulkgridClient } from '@bulkgrid/sdk';
const client = new BulkgridClient({
apiKey: process.env.BULKGRID_API_KEY!,
baseUrl: 'https://bulkgrid.com',
});
const result = await client.index.search({
collectionId: process.env.BULKGRID_COLLECTION_ID!,
query: 'How are failed webhooks retried?',
limit: 3,
});
for (const passage of result.results) {
console.log(passage.text_content, passage.url);
}FAQ
Frequently Asked Questions
Everything you need to know about sources, collections, and indexed retrieval.
Index maintains a searchable copy of the sources you choose. Bulkgrid handles indexing, storage, and scheduled refreshes, then returns relevant passages with source references when you search. Your application or AI agent uses those passages as context.
Search discovers pages on the public web. Crawl retrieves content from pages you specify. Index maintains a persistent, searchable collection of source content and refreshes it over time. Use Index when your application needs repeated retrieval from a defined set of sources.
Add a website with selected pages or discovery rules, including supported documents linked from its pages. You can also sync Markdown and MDX files from GitHub repositories. Private repositories connect through your workspace’s GitHub App installation. These workflows do not index every source-code file or provide a general file-upload service.
A collection groups searchable content for an application or task. Include or exclude sources, folders, and individual URLs, then search that collection by ID. Collections can combine public and private sources your connection can access. Choosing a collection narrows the search; it does not grant additional permissions.
A public Bulkgrid source is a shared index that workspaces subscribe to. A private source belongs to its workspace. This visibility describes the indexed content, not whether the original website is publicly reachable. Removing a public source from your workspace removes your subscription without deleting it for others.
Each source refreshes on its configured schedule. You can also request a manual refresh and inspect the run’s progress and changes. New or changed content becomes searchable after the refresh and indexing finish. Available refresh frequencies depend on your plan and source type; updates are not real-time monitoring.
Index returns ranked passages with source URLs, document identifiers, and metadata in JSON. It combines keyword and semantic search to find relevant context. Your application or agent uses that context to produce an answer and cite its sources.
Yes. Scope API keys or OAuth connections to the collections they should access. Retrieval respects the connection’s existing source and collection permissions. A query’s collection or source filters can narrow results further, but cannot expand that access.
Yes. Try Index in the Playground to search public indexed content. Free workspaces can subscribe to existing public sources and search them. Creating new sources or collections requires a paid plan. See Pricing for current allowances.
Make the Web Work for Your Agents.
Search, crawl, and extract the web. Give your agents the content they need to get to work.
