SDK

Choose TypeScript for Node.js or Python for synchronous and asynchronous Python applications.

TypeScript

Use the TypeScript SDK from your Node.js application or worker.

Install

npm install @bulkgrid/sdk@^0.2.0

Create a key with the scopes your workflow needs, then configure the client:

import { BulkgridClient } from '@bulkgrid/sdk';

const client = new BulkgridClient({
  apiKey: process.env.BULKGRID_API_KEY ?? '',
  baseUrl: process.env.BULKGRID_BASE_URL ?? 'https://bulkgrid.com',
});

The SDK uses API-key authentication. Do not pass an OAuth token as apiKey.

Search indexed content

Requires search:query and accessible indexed content:

const response = await client.search({ query: 'authentication setup', limit: 5 });
console.log(response.results);

Find a source and add it to a collection

Requires sources:read, sources:write, collections:read, collections:write, and search:query. Plan limits and collection restrictions still apply; a 403 is not a reason to retry with broader credentials automatically.

// Search the public catalog and sources owned by your workspace.
const { sources: matches } = await client.sources.search('https://example.com');
console.log(
  matches.map(source => ({
    identifier: source.identifier,
    subscribed: source.isSubscribed,
    owned: source.isOwned,
  })),
);

// For a website, create it or subscribe to the existing shared source.
const { source, created } = await client.sources.create({
  type: 'domain',
  identifier: 'https://example.com',
  visibility: 'public',
});
console.log({ sourceId: source.id, created });

const { collection } = await client.collections.create({ name: 'Research' });
await client.collections.addSource({ collectionId: collection.id, sourceId: source.id });

const status = await client.sources.status(source.id);
if (status.indexedItems > 0) {
  const response = await client.search({ query: 'authentication setup', collectionId: collection.id });
  console.log(response.executedQueries, response.results);
}

created: false means the source already existed. Adding a subscription does not guarantee indexing is complete. sources.list({ search }) searches your subscribed and owned sources; sources.search(query) searches the public catalog and owned sources, with subscription flags. Collection creation requires a qualifying plan. To use an existing collection, call collections.list() and select an appropriate accessible collection instead. addSource includes the whole source and keeps existing exclusions. Use updateRules for specific folders or URLs:

await client.collections.updateRules(collection.id, {
  add: [{ action: 'exclude', target: 'folder', source_id: source.id, target_id: '/archive' }],
});
const { rules, counts } = await client.collections.rules(collection.id);
console.log(rules, counts);

removeSource({ collectionId, sourceId }) removes the source’s rules returned by an initial read. It requires both collections:read and collections:write; rules added concurrently after that read are left alone. Removing a source from a collection does not remove its workspace subscription.

GitHub sources and indexing

const { source: githubSource, run } = await client.sources.createGithub({
  repositoryUrl: 'https://github.com/example/docs',
  includePaths: ['docs/**'],
});
console.log(githubSource.id, run.id);

const page = await client.sources.documents(githubSource.id, { parent: '/', limit: 50 });
console.log(page.documents, page.nextOffset);

GitHub sources require a qualifying plan and are private to the workspace. Source creation returns before ingestion completes. sources.recrawl(id) requests a refresh and needs sources:write plus runs:write. sources.changes(id, { runId, kind: 'changed' }) lists changes for a specific run; paginate with offset and the returned nextOffset.

Create and monitor a crawl

Requires runs:write, runs:read, and results:read:

const run = await client.crawl({
  urls: ['https://example.com'],
  options: { formats: ['markdown'] },
});

const current = await client.runs.get(run.id);
console.log(current.status);

Poll until completed, failed, or cancelled, with a maximum wait time. Once complete:

const { results } = await client.runs.results(run.id, { page: 1, limit: 20 });
const firstResult = results.find(result => !result.error_message);
if (firstResult) {
  const markdown = await client.runs.getResultContent(run.id, firstResult.id, { type: 'markdown' });
  console.log(markdown);
}

Available methods

Method	Purpose
`search(input)`	Search indexed content
`crawl(input)`	Crawl known URLs
`deepCrawl(input)`	Traverse from a starting URL
`extract(input)`	Extract structured fields
`runs.create(input)`	Create a run with an explicit `type`
`runs.list()` / `runs.get(id)`	Inspect runs
`runs.cancel(id)` / `runs.retry(id)`	Stop or retry work
`runs.delete(id)`	Request asynchronous deletion of a stopped run
`runs.results(id)`	List result records
`runs.getResultContent(...)`	Retrieve text content
`runs.getResultScreenshot(...)`	Retrieve a signed screenshot URL

crawl, deepCrawl, and extract add the required run type automatically. Raw HTTP calls and runs.create must include it explicitly.

Source methods	Collection methods
`sources.list(query?)`, `sources.search(query)`	`collections.list()`
`sources.create(input)`, `sources.createGithub(input)`	`collections.create(input)`
`sources.get(id)`, `sources.update(id, input)`	`collections.get(id)`, `collections.update(id, input)`
`sources.delete(id)`	`collections.delete(id)`
`sources.status(id)`, `sources.recrawl(id)`	`collections.addSource({ collectionId, sourceId })`
`sources.documents(id, query?)`, `sources.changes(id, query)`	`collections.removeSource({ collectionId, sourceId })`
	`collections.rules(id)`, `collections.updateRules(id, input)`

sources.delete removes your subscription to a shared source or deletes an owned source according to server permissions. See Source management.

Handle failures

import { BulkgridApiError } from '@bulkgrid/sdk';

try {
  await client.search({ query: 'release notes' });
} catch (error) {
  if (error instanceof BulkgridApiError) {
    console.error(error.status, error.data);
  }
  throw error;
}

Use RequestOptions.signal to cancel requests. The SDK does not automatically retry requests or poll runs. See Retries before retrying a create request that may already have succeeded.

Analyze a source before ingestion

Source analysis is asynchronous and may start a resource-consuming crawl. It requires sources:discover and runs:write; start it only within the approved source scope. These methods are available in the repository build; verify your installed package includes them.

const analysis = await client.sources.analyze({ url: 'https://example.com' });
console.log(analysis.id, analysis.runId, analysis.status);

// Check later using the analysis ID, not the crawl run ID.
const progress = await client.sources.getAnalysis(analysis.id);
console.log(progress.status, progress.urlCount, progress.paths, progress.warning);

Reading analysis requires sources:discover. Counts describe estimated coverage, not guaranteed storage or credits. Treat partial results and warnings as incomplete evidence. The SDK does not poll indefinitely or create a source subscription automatically. After approved creation, use sources.status(sourceId) to distinguish source registration from indexing readiness. Use collections.rules(collectionId) before adding whole-source access with collections.addSource({ sourceId, collectionId }). A failed collection update does not undo source creation; preserve the returned source ID rather than repeating creation.

Python

The Python SDK supports Python 3.10+ and offers the same source, collection, crawl, extraction, search, and run operations as the TypeScript SDK.

The Python package is prepared for its first PyPI release. Until it is published, install it from a local repository checkout with python -m pip install ./packages/sdk-python. After publication, use python -m pip install bulkgrid.

Search

import os
from bulkgrid import BulkgridClient

with BulkgridClient(api_key=os.environ["BULKGRID_API_KEY"]) as client:
    response = client.search({"query": "authentication setup", "limit": 5})
    print(response["results"])

Use a scoped API key. The default base_url is https://bulkgrid.com; you can override it for your deployment. Request and response dictionaries retain the API’s field spelling, such as collectionId and source_id. Method names use Python conventions: deep_crawl, sources.get_analysis, sources.create_github, and runs.get_result_content.

Async applications

import asyncio
import os
from bulkgrid import AsyncBulkgridClient


async def main():
    async with AsyncBulkgridClient(api_key=os.environ["BULKGRID_API_KEY"]) as client:
        response = await client.search({"query": "release notes", "limit": 3})
        print(response["results"])


asyncio.run(main())

Every sync operation has an async equivalent. Use context managers to release HTTP connections, or call close() / await aclose() explicitly.

Types and errors

Request and response types are available from bulkgrid.types. Responses remain regular Python dictionaries; type hints support editor completion and static checking. The server validates requests; the SDK checks HTTP status and JSON object responses. Catch BulkgridAPIError for non-success HTTP responses. It exposes status_code, data, request_id, and retry_after. BulkgridTimeoutError and BulkgridConnectionError identify network failures; BulkgridResponseError identifies invalid JSON responses. The default network timeout is 60 seconds; configure it with timeout=30.0, for example. It does not wait for a crawl or extraction run to finish. Requests do not retry automatically: a timed-out write may already have started processing. Persist run and analysis IDs, use bounded polling, and inspect status before retrying.

Sources, collections, and runs

The Sources, Collections, and Runs and results guides include Python tabs. For example, analyze a source with client.sources.analyze({"url": "https://example.com"}) and check it with client.sources.get_analysis(analysis_id). Use client.collections.add_source(collection_id, source_id) to include a source in a collection. remove_source(collection_id, source_id) removes all rules for that source, matching the TypeScript helper. client.runs.results(run_id, {"page": 1, "limit": 20}) returns one page of results. Text content comes from get_result_content; screenshot URLs come from get_result_screenshot, and download_result_screenshot returns image bytes. Check per-result failures and pagination metadata before consuming outputs.

Get started

Index

More

Access and operations

TypeScript

Install

Search indexed content

Find a source and add it to a collection

GitHub sources and indexing

Create and monitor a crawl

Available methods

Handle failures

Analyze a source before ingestion

Python

Search

Async applications

Types and errors

Sources, collections, and runs

​TypeScript

​Install

​Search indexed content

​Find a source and add it to a collection

​GitHub sources and indexing

​Create and monitor a crawl

​Available methods

​Handle failures

​Analyze a source before ingestion

​Python

​Search

​Async applications

​Types and errors

​Sources, collections, and runs

TypeScript

Install

Search indexed content

Find a source and add it to a collection

GitHub sources and indexing

Create and monitor a crawl

Available methods

Handle failures

Analyze a source before ingestion

Python

Search

Async applications

Types and errors

Sources, collections, and runs