Website Indexing

Source Indexing

Turn websites and public or private GitHub repositories into searchable knowledge for your AI. Choose the content you need, and let Bulkgrid handle crawling, syncing, and indexing.

WebsitesGitHub repositoriesPrivate sourcesLinked documents

Capabilities

Source Indexing Features

Index selected website content, linked documents, and GitHub repository files for retrieval.

Website crawling

Discover pages through links and sitemaps, with browser rendering for JavaScript-driven websites.

GitHub repositories

Index content from public repositories or connect a GitHub App to access permitted private repositories.

Choose your content

Select website pages or repository paths to focus your source on the knowledge your AI needs.

Linked documents

Process supported documents linked from your website, including PDFs and Office files, into searchable text.

Useful structure

Preserve headings, sections, and source references so retrieved passages retain their context.

Private workspace knowledge

Keep private repository content scoped to your workspace and available through authorized access.

Build Your AI’s Knowledge From Sources You Trust

FAQ

Frequently Asked Questions

Practical answers to help you get started.

Yes. Add an existing public source to your workspace and search its indexed content. Creating your own public or private sources requires Pro.

A workspace admin connects the GitHub App and grants access to selected repositories. Private source content stays private to that workspace.

This capability covers supported documents discovered through website crawling. It does not imply a standalone upload workflow or support for every file format.

Get Started With Source Indexing

Start with an existing public source or upgrade to index your own websites and repositories.

Get Started