> ## Documentation Index
> Fetch the complete documentation index at: https://bulkgrid.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Sources

> Add, analyze, and refresh website and GitHub sources.

A source is a persistent content origin with documents, indexing state, and refresh history. Sources can represent websites or [GitHub repositories](/docs/sources).

## Shared and private sources

| Source            | Ownership                                                  | Removing it from a workspace                                          |
| ----------------- | ---------------------------------------------------------- | --------------------------------------------------------------------- |
| Public website    | Shared across workspaces; your workspace subscribes to it  | Removes your subscription, not the shared source for other workspaces |
| Private source    | Owned by your workspace                                    | Removes the workspace-owned source and initiates associated cleanup   |
| GitHub repository | Workspace-owned, even when the repository itself is public | Uses the private-source removal flow                                  |

A public source can reuse existing indexed content. Your workspace's subscription controls its relationship to that source; it does not make the global source your private copy. Use collection rules and credentials to control retrieval access.

For domain sources, identifiers normalize to the URL origin. `https://docs.example.com/foo` and `https://docs.example.com/bar` identify the same source root; choose path filters to define a site section.

## Analyze a website

Source analysis inspects candidate paths and estimates URL coverage. It can start a crawl run and consume resources; authorize its scope before starting it. The standalone synchronous discovery endpoint has been removed.

### Start analysis

Requires `sources:discover` and `runs:write`:

Node.js and Python examples use the [Bulkgrid SDKs](/docs/sdk). Set `BULKGRID_API_KEY` in your backend environment; cURL examples also use `BULKGRID_BASE_URL=https://bulkgrid.com`. CLI examples require the [Bulkgrid CLI](/docs/cli) and an API key with the scopes listed below; browser login does not grant source-management permissions. Set `SOURCE_ID`, `ANALYSIS_ID`, and `RUN_ID` from returned IDs as needed.

<CodeGroup>
  ```js Node.js theme={null}
  import { BulkgridClient } from '@bulkgrid/sdk';

  const client = new BulkgridClient({
    apiKey: process.env.BULKGRID_API_KEY ?? '',
    baseUrl: process.env.BULKGRID_BASE_URL ?? 'https://bulkgrid.com',
  });

  const data = await client.sources.analyze({
    url: 'https://example.com',
  });
  console.log(data);
  ```

  ```python Python theme={null}
  import os
  from bulkgrid import BulkgridClient

  with BulkgridClient(
      api_key=os.environ["BULKGRID_API_KEY"],
      base_url=os.environ.get("BULKGRID_BASE_URL", "https://bulkgrid.com"),
  ) as client:
      data = client.sources.analyze({"url": "https://example.com"})
      print(data)
  ```

  ```bash cURL theme={null}
  curl "$BULKGRID_BASE_URL/api/v1/sources/discover/analyses" \
    -H 'Content-Type: application/json' \
    -H "x-api-key: $BULKGRID_API_KEY" \
    -d '{"url":"https://example.com"}'
  ```

  ```bash CLI theme={null}
  bulkgrid sources analyze https://example.com --url "${BULKGRID_BASE_URL:-https://bulkgrid.com}"
  ```
</CodeGroup>

Save the returned analysis `id` and any `runId`. Analysis does not create a source subscription. The analysis endpoints retain their existing paths under `/sources/discover/analyses`.

### Read progress and results

Requires `sources:discover`:

<CodeGroup>
  ```js Node.js theme={null}
  import { BulkgridClient } from '@bulkgrid/sdk';

  const client = new BulkgridClient({
    apiKey: process.env.BULKGRID_API_KEY ?? '',
    baseUrl: process.env.BULKGRID_BASE_URL ?? 'https://bulkgrid.com',
  });

  const data = await client.sources.getAnalysis(process.env.ANALYSIS_ID ?? '');
  console.log(data);
  ```

  ```python Python theme={null}
  import os
  from bulkgrid import BulkgridClient

  with BulkgridClient(
      api_key=os.environ["BULKGRID_API_KEY"],
      base_url=os.environ.get("BULKGRID_BASE_URL", "https://bulkgrid.com"),
  ) as client:
      data = client.sources.get_analysis(os.environ["ANALYSIS_ID"])
      print(data)
  ```

  ```bash cURL theme={null}
  curl "$BULKGRID_BASE_URL/api/v1/sources/discover/analyses/$ANALYSIS_ID" \
    -H "x-api-key: $BULKGRID_API_KEY"
  ```

  ```bash CLI theme={null}
  bulkgrid sources analysis-status "$ANALYSIS_ID" --url "${BULKGRID_BASE_URL:-https://bulkgrid.com}"
  ```
</CodeGroup>

States are `crawling`, `analyzing`, `completed`, `partial`, and `failed`. Results include the URL, estimated URL count, discovered paths, level summaries, warnings, and errors. Counts may be unavailable while work is in progress. `partial` indicates incomplete coverage; estimates are not guaranteed storage or credit quotes.

Use bounded polling and preserve the ID if the wait expires. Do not blindly resubmit a start request after a timeout: work may already have begun.

### Choose ingestion boundaries

Use the analysis to select relevant paths or pages before creating a source. The path recommendation flow can help choose include/exclude rules; it is distinct from starting analysis and from source creation.

## Create or subscribe to a website source

Requires `sources:write`. Plan limits apply: Free can subscribe to existing public sources, while creating a new public source requires Pro. Private-source creation also requires Pro. Source counts are uncapped; processing, retrieval, storage, and concurrency allowances still apply.

<CodeGroup>
  ```js Node.js theme={null}
  import { BulkgridClient } from '@bulkgrid/sdk';

  const client = new BulkgridClient({
    apiKey: process.env.BULKGRID_API_KEY ?? '',
    baseUrl: process.env.BULKGRID_BASE_URL ?? 'https://bulkgrid.com',
  });

  const data = await client.sources.create({
    type: 'domain',
    identifier: 'https://example.com',
    visibility: 'public',
    source_mode: 'discover',
    crawl_config: {
      includePaths: ['/docs'],
    },
  });
  console.log(data);
  ```

  ```python Python theme={null}
  import os
  from bulkgrid import BulkgridClient

  with BulkgridClient(
      api_key=os.environ["BULKGRID_API_KEY"],
      base_url=os.environ.get("BULKGRID_BASE_URL", "https://bulkgrid.com"),
  ) as client:
      data = client.sources.create(
          {
              "type": "domain",
              "identifier": "https://example.com",
              "visibility": "public",
              "source_mode": "discover",
              "crawl_config": {"includePaths": ["/docs"]},
          }
      )
      print(data)
  ```

  ```bash cURL theme={null}
  curl "$BULKGRID_BASE_URL/api/v1/sources" \
    -H 'Content-Type: application/json' \
    -H "x-api-key: $BULKGRID_API_KEY" \
    -d '{
      "type": "domain",
      "identifier": "https://example.com",
      "visibility": "public",
      "source_mode": "discover",
      "crawl_config": { "includePaths": ["/docs"] }
    }'
  ```

  ```bash CLI theme={null}
  bulkgrid sources add --input '{
    "type": "domain",
    "identifier": "https://example.com",
    "visibility": "public",
    "source_mode": "discover",
    "crawl_config": {
      "includePaths": [
        "/docs"
      ]
    }
  }' --url "${BULKGRID_BASE_URL:-https://bulkgrid.com}"
  ```
</CodeGroup>

The response contains `source` and `created`. Save `source.id`. `created: false` can indicate reuse of an existing source. Creating the source record is not proof that documents are indexed; inspect [source status](/docs/sources).

## Choose the content boundary

* `discover` discovers content using the source's path configuration.
* `selected_pages` processes explicitly selected document URLs. Store them in `crawl_config.selectedUrls`.
* Use collection rules to curate the content each application can retrieve.

Website subscriptions can have workspace-specific labels, selections, and path filters. Changes to shared crawl options can affect the shared source and are plan-gated; they are distinct from selecting content for a collection.

## Read, update, and remove

| Method and path                           | Scope                         | Purpose                                                |
| ----------------------------------------- | ----------------------------- | ------------------------------------------------------ |
| `GET /api/v1/sources`                     | `sources:read`                | List accessible sources; response contains `sources`   |
| `GET /api/v1/sources/{sourceId}`          | `sources:read`                | Read a source in the current workspace                 |
| `PATCH /api/v1/sources/{sourceId}`        | `sources:write`               | Update supported configuration                         |
| `DELETE /api/v1/sources/{sourceId}`       | `sources:delete`              | Remove ownership or subscription as described above    |
| `POST /api/v1/sources/{sourceId}/recrawl` | `sources:write`, `runs:write` | Request a manual refresh; paid-plan restrictions apply |

See [documents and changes](#documents-and-changes) to inspect indexed content and [Collections](/docs/collections) to control retrieval.

## GitHub repositories

GitHub sources synchronize repository content directly rather than crawling GitHub's website. Repository sources require Pro and are private to the Bulkgrid workspace, even when the repository is public.

### Create a source

Requires `sources:write`. Use a public repository URL in the form `https://github.com/{owner}/{repo}`.

<CodeGroup>
  ```js Node.js theme={null}
  import { BulkgridClient } from '@bulkgrid/sdk';

  const client = new BulkgridClient({
    apiKey: process.env.BULKGRID_API_KEY ?? '',
    baseUrl: process.env.BULKGRID_BASE_URL ?? 'https://bulkgrid.com',
  });

  const data = await client.sources.createGithub({
    repositoryUrl: 'https://github.com/microsoft/TypeScript',
    includePaths: ['docs'],
    excludePaths: [],
    crawl_interval: 'weekly',
  });
  console.log(data);
  ```

  ```python Python theme={null}
  import os
  from bulkgrid import BulkgridClient

  with BulkgridClient(
      api_key=os.environ["BULKGRID_API_KEY"],
      base_url=os.environ.get("BULKGRID_BASE_URL", "https://bulkgrid.com"),
  ) as client:
      data = client.sources.create_github(
          {
              "repositoryUrl": "https://github.com/microsoft/TypeScript",
              "includePaths": ["docs"],
              "excludePaths": [],
              "crawl_interval": "weekly",
          }
      )
      print(data)
  ```

  ```bash cURL theme={null}
  curl "$BULKGRID_BASE_URL/api/v1/sources/github" \
    -H 'Content-Type: application/json' \
    -H "x-api-key: $BULKGRID_API_KEY" \
    -d '{
      "repositoryUrl": "https://github.com/microsoft/TypeScript",
      "includePaths": ["docs"],
      "excludePaths": [],
      "crawl_interval": "weekly"
    }'
  ```
</CodeGroup>

The server returns `202` with `source`, `run`, and `dispatchDeferred`. A deferred dispatch means the job was saved and is awaiting dispatch; avoid creating a duplicate source. Creating the same repository again in the workspace returns `409`.

### Configuration

| Field                          | Purpose                                                          |
| ------------------------------ | ---------------------------------------------------------------- |
| `repositoryUrl`                | Required public GitHub repository URL                            |
| `ref`                          | Optional repository ref; use the repository default when omitted |
| `includePaths`, `excludePaths` | Repository path filters                                          |
| `label`                        | Optional display name                                            |
| `canonical_url`                | Optional related website URL for source identity and preview     |
| `crawl_interval`               | `daily`, `weekly`, or `monthly`; default `weekly`                |

A canonical website URL does not replace the repository as the source of indexed content. The website preview can complete independently of repository synchronization.

### Monitor and refresh

Repository runs have `type: "source_sync"`. Poll the run and source status, and inspect indexed document counts before using search.

Use `POST /api/v1/sources/{sourceId}/recrawl` for a manual sync. It requires `sources:write` and `runs:write`, plus an active paid plan. For GitHub sources this starts repository synchronization, not a website deep crawl.

Add the source to a collection and authorize that collection for your application or AI client. See [Collections](/docs/collections).

## Refresh and monitor

### Request a manual refresh

Requires `sources:write` and `runs:write`. Manual refresh requires an active or trialing paid plan; otherwise the API returns `403` with `LIMIT_MANUAL_REINDEX`.

<CodeGroup>
  ```js Node.js theme={null}
  import { BulkgridClient } from '@bulkgrid/sdk';

  const client = new BulkgridClient({
    apiKey: process.env.BULKGRID_API_KEY ?? '',
    baseUrl: process.env.BULKGRID_BASE_URL ?? 'https://bulkgrid.com',
  });

  const data = await client.sources.recrawl(process.env.SOURCE_ID ?? '');
  console.log(data);
  ```

  ```python Python theme={null}
  import os
  from bulkgrid import BulkgridClient

  with BulkgridClient(
      api_key=os.environ["BULKGRID_API_KEY"],
      base_url=os.environ.get("BULKGRID_BASE_URL", "https://bulkgrid.com"),
  ) as client:
      data = client.sources.recrawl(os.environ["SOURCE_ID"])
      print(data)
  ```

  ```bash cURL theme={null}
  curl -X POST "$BULKGRID_BASE_URL/api/v1/sources/$SOURCE_ID/recrawl" \
    -H "x-api-key: $BULKGRID_API_KEY"
  ```

  ```bash CLI theme={null}
  bulkgrid sources recrawl "$SOURCE_ID" --url "${BULKGRID_BASE_URL:-https://bulkgrid.com}"
  ```
</CodeGroup>

The response identifies the source and run. Inspect the returned run rather than assuming a refresh has already completed.

| Source                    | Refresh behavior                                                                                |
| ------------------------- | ----------------------------------------------------------------------------------------------- |
| Website, `discover`       | Deep crawl using the source's crawl configuration                                               |
| Website, `selected_pages` | Crawl the selected document URLs; the source identifier is the fallback when no URLs are stored |
| GitHub repository         | A `source_sync` run for repository content                                                      |

A shared source remains shared when refreshed. Refreshing it does not create a private copy for the requesting workspace.

### Scheduled refresh

Website source configuration supports `crawl_interval` and, for `custom`, `custom_interval_minutes`. GitHub creation supports `daily`, `weekly`, and `monthly`. Available frequencies depend on the plan; handle returned plan-limit errors rather than assuming every interval is available.

### Check source status

Requires `sources:read`:

<CodeGroup>
  ```js Node.js theme={null}
  import { BulkgridClient } from '@bulkgrid/sdk';

  const client = new BulkgridClient({
    apiKey: process.env.BULKGRID_API_KEY ?? '',
    baseUrl: process.env.BULKGRID_BASE_URL ?? 'https://bulkgrid.com',
  });

  const data = await client.sources.status(process.env.SOURCE_ID ?? '');
  console.log(data);
  ```

  ```python Python theme={null}
  import os
  from bulkgrid import BulkgridClient

  with BulkgridClient(
      api_key=os.environ["BULKGRID_API_KEY"],
      base_url=os.environ.get("BULKGRID_BASE_URL", "https://bulkgrid.com"),
  ) as client:
      data = client.sources.status(os.environ["SOURCE_ID"])
      print(data)
  ```

  ```bash cURL theme={null}
  curl "$BULKGRID_BASE_URL/api/v1/sources/$SOURCE_ID/status" \
    -H "x-api-key: $BULKGRID_API_KEY"
  ```

  ```bash CLI theme={null}
  bulkgrid sources status "$SOURCE_ID" --url "${BULKGRID_BASE_URL:-https://bulkgrid.com}"
  ```
</CodeGroup>

The status includes `isCrawling`, `latestStatus`, `latestError`, `totalSize`, `totalItems`, and `indexedItems`. Use indexed counts and errors to distinguish ingestion progress from searchable content.

Use `GET /api/v1/sources/{sourceId}/runs` with `runs:read` for history. Use [run status](/docs/runs-and-results) to inspect a particular job and [source documents](#documents-and-changes) to inspect its content.

## Documents and changes

Requires `sources:read` and access to the source through workspace ownership or subscription.

### List documents

<CodeGroup>
  ```js Node.js theme={null}
  import { BulkgridClient } from '@bulkgrid/sdk';

  const client = new BulkgridClient({
    apiKey: process.env.BULKGRID_API_KEY ?? '',
    baseUrl: process.env.BULKGRID_BASE_URL ?? 'https://bulkgrid.com',
  });

  const data = await client.sources.documents(process.env.SOURCE_ID ?? '', {
    parent: '/',
    offset: 0,
    limit: 50,
  });
  console.log(data);
  ```

  ```python Python theme={null}
  import os
  from bulkgrid import BulkgridClient

  with BulkgridClient(
      api_key=os.environ["BULKGRID_API_KEY"],
      base_url=os.environ.get("BULKGRID_BASE_URL", "https://bulkgrid.com"),
  ) as client:
      data = client.sources.documents(
          os.environ["SOURCE_ID"], {"parent": "/", "offset": 0, "limit": 50}
      )
      print(data)
  ```

  ```bash cURL theme={null}
  curl "$BULKGRID_BASE_URL/api/v1/sources/$SOURCE_ID/documents?parent=/&offset=0&limit=50" \
    -H "x-api-key: $BULKGRID_API_KEY"
  ```
</CodeGroup>

The response contains `documents`, `parent`, `nextOffset`, `limit`, and `total`. Continue from `nextOffset` until it is `null`. The default limit is `50`, capped at `100`. Use `parent` to browse a source path; `withTotal` is not required.

Documents include source content identifiers and metadata useful for inspecting crawl and indexing state. A source document ID is distinct from a run result ID.

### Browse folders

Use `GET /api/v1/sources/{sourceId}/folders?parent=/` to browse folders under a path. Use folder or URL [collection rules](/docs/collections) to curate content for retrieval.

### Inspect a run's changes

The `runId` query parameter is required:

<CodeGroup>
  ```js Node.js theme={null}
  import { BulkgridClient } from '@bulkgrid/sdk';

  const client = new BulkgridClient({
    apiKey: process.env.BULKGRID_API_KEY ?? '',
    baseUrl: process.env.BULKGRID_BASE_URL ?? 'https://bulkgrid.com',
  });

  const data = await client.sources.changes(process.env.SOURCE_ID ?? '', {
    runId: process.env.RUN_ID ?? '',
    kind: 'changed',
    offset: 0,
    limit: 50,
  });
  console.log(data);
  ```

  ```python Python theme={null}
  import os
  from bulkgrid import BulkgridClient

  with BulkgridClient(
      api_key=os.environ["BULKGRID_API_KEY"],
      base_url=os.environ.get("BULKGRID_BASE_URL", "https://bulkgrid.com"),
  ) as client:
      data = client.sources.changes(
          os.environ["SOURCE_ID"],
          {"runId": os.environ["RUN_ID"], "kind": "changed", "offset": 0, "limit": 50},
      )
      print(data)
  ```

  ```bash cURL theme={null}
  curl "$BULKGRID_BASE_URL/api/v1/sources/$SOURCE_ID/changes?runId=$RUN_ID&kind=changed&offset=0&limit=50" \
    -H "x-api-key: $BULKGRID_API_KEY"
  ```
</CodeGroup>

`kind` accepts `added`, `changed`, or `removed`, and defaults to `changed`. The default limit is `50`, capped at `500`. The response contains `documents`, `total`, `nextOffset`, and `limit`.

Use `GET /api/v1/sources/{sourceId}/runs` with `runs:read` to find the relevant source run. Changes are scoped to the requested run, not an unspecified latest comparison.