Sources

A source is a persistent content origin with documents, indexing state, and refresh history. Sources can represent websites or GitHub repositories.

Shared and private sources

Source	Ownership	Removing it from a workspace
Public website	Shared across workspaces; your workspace subscribes to it	Removes your subscription, not the shared source for other workspaces
Private source	Owned by your workspace	Removes the workspace-owned source and initiates associated cleanup
GitHub repository	Workspace-owned, even when the repository itself is public	Uses the private-source removal flow

A public source can reuse existing indexed content. Your workspace’s subscription controls its relationship to that source; it does not make the global source your private copy. Use collection rules and credentials to control retrieval access. For domain sources, identifiers normalize to the URL origin. https://docs.example.com/foo and https://docs.example.com/bar identify the same source root; choose path filters to define a site section.

Analyze a website

Source analysis inspects candidate paths and estimates URL coverage. It can start a crawl run and consume resources; authorize its scope before starting it. The standalone synchronous discovery endpoint has been removed.

Start analysis

Requires sources:discover and runs:write: Node.js and Python examples use the Bulkgrid SDKs. Set BULKGRID_API_KEY in your backend environment; cURL examples also use BULKGRID_BASE_URL=https://bulkgrid.com. CLI examples require the Bulkgrid CLI and an API key with the scopes listed below; browser login does not grant source-management permissions. Set SOURCE_ID, ANALYSIS_ID, and RUN_ID from returned IDs as needed.

import { BulkgridClient } from '@bulkgrid/sdk';

const client = new BulkgridClient({
  apiKey: process.env.BULKGRID_API_KEY ?? '',
  baseUrl: process.env.BULKGRID_BASE_URL ?? 'https://bulkgrid.com',
});

const data = await client.sources.analyze({
  url: 'https://example.com',
});
console.log(data);

import os
from bulkgrid import BulkgridClient

with BulkgridClient(
    api_key=os.environ["BULKGRID_API_KEY"],
    base_url=os.environ.get("BULKGRID_BASE_URL", "https://bulkgrid.com"),
) as client:
    data = client.sources.analyze({"url": "https://example.com"})
    print(data)

curl "$BULKGRID_BASE_URL/api/v1/sources/discover/analyses" \
  -H 'Content-Type: application/json' \
  -H "x-api-key: $BULKGRID_API_KEY" \
  -d '{"url":"https://example.com"}'

bulkgrid sources analyze https://example.com --url "${BULKGRID_BASE_URL:-https://bulkgrid.com}"

Save the returned analysis id and any runId. Analysis does not create a source subscription. The analysis endpoints retain their existing paths under /sources/discover/analyses.

Read progress and results

Requires sources:discover:

import { BulkgridClient } from '@bulkgrid/sdk';

const client = new BulkgridClient({
  apiKey: process.env.BULKGRID_API_KEY ?? '',
  baseUrl: process.env.BULKGRID_BASE_URL ?? 'https://bulkgrid.com',
});

const data = await client.sources.getAnalysis(process.env.ANALYSIS_ID ?? '');
console.log(data);

import os
from bulkgrid import BulkgridClient

with BulkgridClient(
    api_key=os.environ["BULKGRID_API_KEY"],
    base_url=os.environ.get("BULKGRID_BASE_URL", "https://bulkgrid.com"),
) as client:
    data = client.sources.get_analysis(os.environ["ANALYSIS_ID"])
    print(data)

curl "$BULKGRID_BASE_URL/api/v1/sources/discover/analyses/$ANALYSIS_ID" \
  -H "x-api-key: $BULKGRID_API_KEY"

bulkgrid sources analysis-status "$ANALYSIS_ID" --url "${BULKGRID_BASE_URL:-https://bulkgrid.com}"

States are crawling, analyzing, completed, partial, and failed. Results include the URL, estimated URL count, discovered paths, level summaries, warnings, and errors. Counts may be unavailable while work is in progress. partial indicates incomplete coverage; estimates are not guaranteed storage or credit quotes. Use bounded polling and preserve the ID if the wait expires. Do not blindly resubmit a start request after a timeout: work may already have begun.

Choose ingestion boundaries

Use the analysis to select relevant paths or pages before creating a source. The path recommendation flow can help choose include/exclude rules; it is distinct from starting analysis and from source creation. Requires sources:write. Plan limits apply: Free can subscribe to existing public sources, while creating a new public source requires Pro. Private-source creation also requires Pro. Source counts are uncapped; processing, retrieval, storage, and concurrency allowances still apply.

import { BulkgridClient } from '@bulkgrid/sdk';

const client = new BulkgridClient({
  apiKey: process.env.BULKGRID_API_KEY ?? '',
  baseUrl: process.env.BULKGRID_BASE_URL ?? 'https://bulkgrid.com',
});

const data = await client.sources.create({
  type: 'domain',
  identifier: 'https://example.com',
  visibility: 'public',
  source_mode: 'discover',
  crawl_config: {
    includePaths: ['/docs'],
  },
});
console.log(data);

import os
from bulkgrid import BulkgridClient

with BulkgridClient(
    api_key=os.environ["BULKGRID_API_KEY"],
    base_url=os.environ.get("BULKGRID_BASE_URL", "https://bulkgrid.com"),
) as client:
    data = client.sources.create(
        {
            "type": "domain",
            "identifier": "https://example.com",
            "visibility": "public",
            "source_mode": "discover",
            "crawl_config": {"includePaths": ["/docs"]},
        }
    )
    print(data)

curl "$BULKGRID_BASE_URL/api/v1/sources" \
  -H 'Content-Type: application/json' \
  -H "x-api-key: $BULKGRID_API_KEY" \
  -d '{
    "type": "domain",
    "identifier": "https://example.com",
    "visibility": "public",
    "source_mode": "discover",
    "crawl_config": { "includePaths": ["/docs"] }
  }'

bulkgrid sources add --input '{
  "type": "domain",
  "identifier": "https://example.com",
  "visibility": "public",
  "source_mode": "discover",
  "crawl_config": {
    "includePaths": [
      "/docs"
    ]
  }
}' --url "${BULKGRID_BASE_URL:-https://bulkgrid.com}"

The response contains source and created. Save source.id. created: false can indicate reuse of an existing source. Creating the source record is not proof that documents are indexed; inspect source status.

Choose the content boundary

discover discovers content using the source’s path configuration.
selected_pages processes explicitly selected document URLs. Store them in crawl_config.selectedUrls.
Use collection rules to curate the content each application can retrieve.

Website subscriptions can have workspace-specific labels, selections, and path filters. Changes to shared crawl options can affect the shared source and are plan-gated; they are distinct from selecting content for a collection.

Read, update, and remove

Method and path	Scope	Purpose
`GET /api/v1/sources`	`sources:read`	List accessible sources; response contains `sources`
`GET /api/v1/sources/{sourceId}`	`sources:read`	Read a source in the current workspace
`PATCH /api/v1/sources/{sourceId}`	`sources:write`	Update supported configuration
`DELETE /api/v1/sources/{sourceId}`	`sources:delete`	Remove ownership or subscription as described above
`POST /api/v1/sources/{sourceId}/recrawl`	`sources:write`, `runs:write`	Request a manual refresh; paid-plan restrictions apply

See documents and changes to inspect indexed content and Collections to control retrieval.

GitHub repositories

GitHub sources synchronize repository content directly rather than crawling GitHub’s website. Repository sources require Pro and are private to the Bulkgrid workspace, even when the repository is public.

Create a source

Requires sources:write. Use a public repository URL in the form https://github.com/{owner}/{repo}.

import { BulkgridClient } from '@bulkgrid/sdk';

const client = new BulkgridClient({
  apiKey: process.env.BULKGRID_API_KEY ?? '',
  baseUrl: process.env.BULKGRID_BASE_URL ?? 'https://bulkgrid.com',
});

const data = await client.sources.createGithub({
  repositoryUrl: 'https://github.com/microsoft/TypeScript',
  includePaths: ['docs'],
  excludePaths: [],
  crawl_interval: 'weekly',
});
console.log(data);

import os
from bulkgrid import BulkgridClient

with BulkgridClient(
    api_key=os.environ["BULKGRID_API_KEY"],
    base_url=os.environ.get("BULKGRID_BASE_URL", "https://bulkgrid.com"),
) as client:
    data = client.sources.create_github(
        {
            "repositoryUrl": "https://github.com/microsoft/TypeScript",
            "includePaths": ["docs"],
            "excludePaths": [],
            "crawl_interval": "weekly",
        }
    )
    print(data)

curl "$BULKGRID_BASE_URL/api/v1/sources/github" \
  -H 'Content-Type: application/json' \
  -H "x-api-key: $BULKGRID_API_KEY" \
  -d '{
    "repositoryUrl": "https://github.com/microsoft/TypeScript",
    "includePaths": ["docs"],
    "excludePaths": [],
    "crawl_interval": "weekly"
  }'

The server returns 202 with source, run, and dispatchDeferred. A deferred dispatch means the job was saved and is awaiting dispatch; avoid creating a duplicate source. Creating the same repository again in the workspace returns 409.

Configuration

Field	Purpose
`repositoryUrl`	Required public GitHub repository URL
`ref`	Optional repository ref; use the repository default when omitted
`includePaths`, `excludePaths`	Repository path filters
`label`	Optional display name
`canonical_url`	Optional related website URL for source identity and preview
`crawl_interval`	`daily`, `weekly`, or `monthly`; default `weekly`

A canonical website URL does not replace the repository as the source of indexed content. The website preview can complete independently of repository synchronization.

Monitor and refresh

Repository runs have type: "source_sync". Poll the run and source status, and inspect indexed document counts before using search. Use POST /api/v1/sources/{sourceId}/recrawl for a manual sync. It requires sources:write and runs:write, plus an active paid plan. For GitHub sources this starts repository synchronization, not a website deep crawl. Add the source to a collection and authorize that collection for your application or AI client. See Collections.

Refresh and monitor

Request a manual refresh

Requires sources:write and runs:write. Manual refresh requires an active or trialing paid plan; otherwise the API returns 403 with LIMIT_MANUAL_REINDEX.

import { BulkgridClient } from '@bulkgrid/sdk';

const client = new BulkgridClient({
  apiKey: process.env.BULKGRID_API_KEY ?? '',
  baseUrl: process.env.BULKGRID_BASE_URL ?? 'https://bulkgrid.com',
});

const data = await client.sources.recrawl(process.env.SOURCE_ID ?? '');
console.log(data);

import os
from bulkgrid import BulkgridClient

with BulkgridClient(
    api_key=os.environ["BULKGRID_API_KEY"],
    base_url=os.environ.get("BULKGRID_BASE_URL", "https://bulkgrid.com"),
) as client:
    data = client.sources.recrawl(os.environ["SOURCE_ID"])
    print(data)

curl -X POST "$BULKGRID_BASE_URL/api/v1/sources/$SOURCE_ID/recrawl" \
  -H "x-api-key: $BULKGRID_API_KEY"

bulkgrid sources recrawl "$SOURCE_ID" --url "${BULKGRID_BASE_URL:-https://bulkgrid.com}"

The response identifies the source and run. Inspect the returned run rather than assuming a refresh has already completed.

Source	Refresh behavior
Website, `discover`	Deep crawl using the source’s crawl configuration
Website, `selected_pages`	Crawl the selected document URLs; the source identifier is the fallback when no URLs are stored
GitHub repository	A `source_sync` run for repository content

A shared source remains shared when refreshed. Refreshing it does not create a private copy for the requesting workspace.

Scheduled refresh

Website source configuration supports crawl_interval and, for custom, custom_interval_minutes. GitHub creation supports daily, weekly, and monthly. Available frequencies depend on the plan; handle returned plan-limit errors rather than assuming every interval is available.

Check source status

Requires sources:read:

import { BulkgridClient } from '@bulkgrid/sdk';

const client = new BulkgridClient({
  apiKey: process.env.BULKGRID_API_KEY ?? '',
  baseUrl: process.env.BULKGRID_BASE_URL ?? 'https://bulkgrid.com',
});

const data = await client.sources.status(process.env.SOURCE_ID ?? '');
console.log(data);

import os
from bulkgrid import BulkgridClient

with BulkgridClient(
    api_key=os.environ["BULKGRID_API_KEY"],
    base_url=os.environ.get("BULKGRID_BASE_URL", "https://bulkgrid.com"),
) as client:
    data = client.sources.status(os.environ["SOURCE_ID"])
    print(data)

curl "$BULKGRID_BASE_URL/api/v1/sources/$SOURCE_ID/status" \
  -H "x-api-key: $BULKGRID_API_KEY"

bulkgrid sources status "$SOURCE_ID" --url "${BULKGRID_BASE_URL:-https://bulkgrid.com}"

The status includes isCrawling, latestStatus, latestError, totalSize, totalItems, and indexedItems. Use indexed counts and errors to distinguish ingestion progress from searchable content. Use GET /api/v1/sources/{sourceId}/runs with runs:read for history. Use run status to inspect a particular job and source documents to inspect its content.

Documents and changes

Requires sources:read and access to the source through workspace ownership or subscription.

List documents

import { BulkgridClient } from '@bulkgrid/sdk';

const client = new BulkgridClient({
  apiKey: process.env.BULKGRID_API_KEY ?? '',
  baseUrl: process.env.BULKGRID_BASE_URL ?? 'https://bulkgrid.com',
});

const data = await client.sources.documents(process.env.SOURCE_ID ?? '', {
  parent: '/',
  offset: 0,
  limit: 50,
});
console.log(data);

import os
from bulkgrid import BulkgridClient

with BulkgridClient(
    api_key=os.environ["BULKGRID_API_KEY"],
    base_url=os.environ.get("BULKGRID_BASE_URL", "https://bulkgrid.com"),
) as client:
    data = client.sources.documents(
        os.environ["SOURCE_ID"], {"parent": "/", "offset": 0, "limit": 50}
    )
    print(data)

curl "$BULKGRID_BASE_URL/api/v1/sources/$SOURCE_ID/documents?parent=/&offset=0&limit=50" \
  -H "x-api-key: $BULKGRID_API_KEY"

The response contains documents, parent, nextOffset, limit, and total. Continue from nextOffset until it is null. The default limit is 50, capped at 100. Use parent to browse a source path; withTotal is not required. Documents include source content identifiers and metadata useful for inspecting crawl and indexing state. A source document ID is distinct from a run result ID.

Browse folders

Use GET /api/v1/sources/{sourceId}/folders?parent=/ to browse folders under a path. Use folder or URL collection rules to curate content for retrieval.

Inspect a run’s changes

The runId query parameter is required:

import { BulkgridClient } from '@bulkgrid/sdk';

const client = new BulkgridClient({
  apiKey: process.env.BULKGRID_API_KEY ?? '',
  baseUrl: process.env.BULKGRID_BASE_URL ?? 'https://bulkgrid.com',
});

const data = await client.sources.changes(process.env.SOURCE_ID ?? '', {
  runId: process.env.RUN_ID ?? '',
  kind: 'changed',
  offset: 0,
  limit: 50,
});
console.log(data);

import os
from bulkgrid import BulkgridClient

with BulkgridClient(
    api_key=os.environ["BULKGRID_API_KEY"],
    base_url=os.environ.get("BULKGRID_BASE_URL", "https://bulkgrid.com"),
) as client:
    data = client.sources.changes(
        os.environ["SOURCE_ID"],
        {"runId": os.environ["RUN_ID"], "kind": "changed", "offset": 0, "limit": 50},
    )
    print(data)

curl "$BULKGRID_BASE_URL/api/v1/sources/$SOURCE_ID/changes?runId=$RUN_ID&kind=changed&offset=0&limit=50" \
  -H "x-api-key: $BULKGRID_API_KEY"

kind accepts added, changed, or removed, and defaults to changed. The default limit is 50, capped at 500. The response contains documents, total, nextOffset, and limit. Use GET /api/v1/sources/{sourceId}/runs with runs:read to find the relevant source run. Changes are scoped to the requested run, not an unspecified latest comparison.

Get started

Index

More

Access and operations

Shared and private sources

Analyze a website

Start analysis

Read progress and results

Choose ingestion boundaries

Choose the content boundary

Read, update, and remove

GitHub repositories

Create a source

Configuration

Monitor and refresh

Refresh and monitor

Request a manual refresh

Scheduled refresh

Check source status

Documents and changes

List documents

Browse folders

Inspect a run’s changes

​Shared and private sources

​Analyze a website

​Start analysis

​Read progress and results

​Choose ingestion boundaries

​Create or subscribe to a website source

​Choose the content boundary

​Read, update, and remove

​GitHub repositories

​Create a source

​Configuration

​Monitor and refresh

​Refresh and monitor

​Request a manual refresh

​Scheduled refresh

​Check source status

​Documents and changes

​List documents

​Browse folders

​Inspect a run’s changes

Shared and private sources

Analyze a website

Start analysis

Read progress and results

Choose ingestion boundaries

Create or subscribe to a website source

Choose the content boundary

Read, update, and remove

GitHub repositories

Create a source

Configuration

Monitor and refresh

Refresh and monitor

Request a manual refresh

Scheduled refresh

Check source status

Documents and changes

List documents

Browse folders

Inspect a run’s changes