> ## Documentation Index
> Fetch the complete documentation index at: https://bulkgrid.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Extraction

> Extract structured data from one or more URLs with asynchronous Bulkgrid runs.

Use extraction when you want fields, not just content.

## When extraction is the right tool

Extraction is a strong fit for:

* company and product profiles
* pricing or policy extraction
* structured enrichment for downstream systems
* repeatable data collection from public pages

## Design the request carefully

Good extraction quality usually depends more on request design than on retry count.

Keep the request:

* narrow enough to be realistic
* specific about what should be extracted
* backed by a schema that downstream systems can actually use

## Generate a schema

Write the JSON schema yourself, as in the examples below, or generate a draft from your extraction goal. Schema generation requires the `extract:schema:generate` scope.

Node.js and Python examples use the [Bulkgrid SDKs](/docs/sdk). Set `BULKGRID_API_KEY` in your backend environment; cURL examples also use `BULKGRID_BASE_URL=https://bulkgrid.com`.

```bash cURL theme={null}
curl "$BULKGRID_BASE_URL/api/v1/extract/schema" \
  -H 'Content-Type: application/json' \
  -H "x-api-key: $BULKGRID_API_KEY" \
  -d '{"query":"Company name and product summary","allowNulls":true}'
```

Schema generation is currently available through the REST API; the SDK has no schema-generation method. The response contains `schema` and optional `notes`. Review the result before passing `schema` to `/api/v1/extract`.

Provide a `query`, optional context, and `allowNulls` to control whether nulls are allowed. Generated schemas are validated before being returned; if validation fails, the service retries once and then returns a structured error.

Review and tighten the draft, then test it with a small extraction run before expanding its scope. Schema generation prepares the request; submit the extraction separately to process your URLs.

## Request examples

<CodeGroup>
  ```js Node.js theme={null}
  import { BulkgridClient } from '@bulkgrid/sdk';

  const client = new BulkgridClient({
    apiKey: process.env.BULKGRID_API_KEY ?? '',
    baseUrl: process.env.BULKGRID_BASE_URL ?? 'https://bulkgrid.com',
  });

  const run = await client.extract({
    urls: ['https://example.com', 'https://example.com/pricing'],
    query: 'Extract company name, product summary, and pricing details',
    schema: {
      type: 'object',
      properties: {
        companyName: { type: 'string' },
        productSummary: { type: 'string' },
        pricing: { type: 'string' },
      },
      required: ['companyName'],
    },
    maxRetries: 3,
  });
  ```

  ```python Python theme={null}
  import os
  from bulkgrid import BulkgridClient

  with BulkgridClient(
      api_key=os.environ["BULKGRID_API_KEY"],
      base_url=os.environ.get("BULKGRID_BASE_URL", "https://bulkgrid.com"),
  ) as client:
      data = client.extract(
          {
              "urls": ["https://example.com", "https://example.com/pricing"],
              "query": "Extract company name, product summary, and pricing details",
              "schema": {
                  "type": "object",
                  "properties": {
                      "companyName": {"type": "string"},
                      "productSummary": {"type": "string"},
                      "pricing": {"type": "string"},
                  },
                  "required": ["companyName"],
              },
              "maxRetries": 3,
          }
      )
      print(data)
  ```

  ```bash cURL theme={null}
  curl "$BULKGRID_BASE_URL/api/v1/extract" \
    -H 'Content-Type: application/json' \
    -H "x-api-key: $BULKGRID_API_KEY" \
    -d '{
      "type": "extract",
      "urls": [
        "https://example.com",
        "https://example.com/pricing"
      ],
      "query": "Extract company name, product summary, and pricing details",
      "schema": {
        "type": "object",
        "properties": {
          "companyName": { "type": "string" },
          "productSummary": { "type": "string" },
          "pricing": { "type": "string" }
        },
        "required": ["companyName"]
      },
      "maxRetries": 3
    }'
  ```
</CodeGroup>

## Workflow

1. submit the extraction request
2. store the run ID
3. poll `GET /api/v1/runs/{runId}`
4. fetch `GET /api/v1/runs/{runId}/results`
5. read `extraction_data` from the result records

## Common quality problems

* the schema asks for data the source does not contain
* the query is too broad
* the page requires interaction or access patterns the request does not account for

## Recommendation

Start with the smallest schema that delivers value. Expand later once the output is stable.