Data Loaders

A Data Loader is a saved import that turns source content into BotDojo documents. The source can be a website, BotDojo flow history, a BotDojo-managed repo, or a connected external system. The documents land in one project folder; indexes can then read that folder.

Records become documents

For an integration-backed Data Loader, each source record becomes a BotDojo document. A record might be a Notion page or database row, a Jira issue, a Zendesk ticket, a Slack message thread, a Google Drive file, or an object from another connected system.

The stored document has two complementary parts:

• Content is the readable text that BotDojo chunks and embeds for semantic search.
• JSON metadata contains attributes and related information about the record, such as its source ID, URL, status, owner, timestamps, tags, project, channel, or provider-specific fields.

Loaders preserve a stable source identifier so a later run can update the same document instead of creating a duplicate. Depending on the loader, source attributes may be written directly to the document’s meta object or included as YAML front matter in a Markdown document. When the document is indexed, those attributes become metadata on its searchable parts.

This separation lets an agent find a record by meaning and then narrow the results by exact attributes. For example, an agent can search for a customer’s refund problem while filtering to open support records from a particular region.

See Indexes for the complete document-to-query path, Metadata and Markdown for front matter and JSON metadata, and Materialized Columns for large-index filtering performance.

Create a Data Loader

1. Open Data in a project.
2. Select Data Loaders.
3. Select Create Loader.
4. Choose a source.

⤢Expand

Screenshot previewThe Data Loader source picker

×

Choose the loader type

BotDojo-native loaders

Native loaders do not require an external connection. Each one addresses a different BotDojo data source:

Loader	Use it for	Important choices
Scrape Website	Crawl a website and turn its pages into documents. Use it for public documentation, help centers, marketing sites, or other web content that should be searchable in BotDojo.	Source and starting URLs, Wget or Playwright, crawl limits, content selectors, URL filters, and deletion behavior.
BotDojo Flow	Turn a flow’s conversation history into Markdown documents for search, reporting, evaluation, or analysis. The source picker currently labels this option Flow Document Loader. It exports conversation data, not the flow definition itself.	Flow, grouping by Session or Thread, date range, flow Version, and Channels.
BotDojo Repo	Keep files from a BotDojo-managed repo available as project documents. Use it for repo-backed documentation, skill and workflow source, or other maintained files that agents should retrieve.	Repo, optional Branch, optional Path Filter, and Run on Commit for automatic updates after commits to the selected branch.

For every scraper field and crawl control, see Web Scraping.

Integration-backed loaders

Integration-backed loaders import content from systems outside BotDojo, such as Google Drive, Notion, S3, SharePoint, GitHub, Zendesk, Slack, Jira, or Confluence. Use one when the source system should remain the owner of the content and BotDojo should synchronize a searchable copy.

A CONNECTED label means the project already has a compatible connection. If a provider is not connected, BotDojo guides you through creating or selecting a connection before configuring the loader. The provider determines what can be selected or filtered—for example, a Google Drive folder, Slack channels, a Jira project, or an S3 location.

The source-specific portion of the form changes by provider. For example, a Google Drive loader asks which drive or folder to sync, while a web scraper asks for a URL and parser. Every loader also has the common settings below.

⤢Expand

Screenshot previewCommon Data Loader settings, shown with the web scraper

×

Common settings

Setting	What it does
Name	Identifies the loader in the list and run history.
Integration	For integration-backed loaders, selects the connection used to read the source. Native loaders do not require one. It cannot normally be changed after creation.
Source settings	Defines what the provider should read. The fields vary by provider and may include a folder picker, query, project, channel, content type, or other source filters.
Destination Folder	The project folder where synchronized documents are written. Select Browse Folders to choose or create it.
Update Schedule	Controls whether the loader runs only when requested or on a recurring schedule.
Failure notification recipients	Optional comma- or semicolon-separated email addresses. Leave blank to use the project-level recipients.
Run Now	Starts the first synchronization after the loader is saved. Clear it when you want to save the configuration without immediately reading the source.

Changing source settings or the destination folder resets the loader’s last-success marker so the next run can establish a fresh synchronization state.

Schedule settings

The schedule uses the time zone reported by your browser when you save the loader.

Schedule	Additional setting	Behavior
Manual	None	Runs only when you select Run Data Loader or Run Loader.
Interval	30 minutes, 1 hour, 3 hours, 6 hours, or 12 hours	Repeats after the chosen interval.
Daily	Schedule Time	Runs once each day at the selected time.
Weekly	One or more weekdays and Schedule Time	Runs on the selected days at the selected time. At least one day is required.
Monthly	Date and Schedule Time	Runs on the selected day of each month.
Custom	Cron Value	Uses a cron expression for schedules not covered by the presets.

The Data Loaders list

The list provides these columns:

Column	Meaning
Name	Opens the loader and its run history.
Provider	The source type or connection provider.
Folder	The destination folder that receives documents.
Last Run	Date of the most recent successful run.
Schedule	Human-readable recurring schedule or Manual.
Next Run	Next calculated scheduled run.
Status	Current loader health.

Health filters appear when relevant:

• Running means a loader job is queued, running, or aborting.
• Needs attention means the most recent run has an error or requires user action.
• Not run means the loader is new or has no successful run.
• A loader without one of those conditions is ready.

Select one or more rows to use Run Data Loader or Archive Data Loader. BotDojo prevents a second run while the same loader already has an active job.

Loader details and run history

Select a loader name to open its detail page. The header shows the destination folder, provider, schedule, and selected source details. Use:

Action	What it does
Edit Loader	Updates the name, source-specific options, destination, schedule, notifications, or Run Now choice.
Run Loader	Starts an ad hoc synchronization without changing the saved schedule.

The Runs table records:

• Status and job progress
• Job type
• Total documents created
• Start and finish times
• Duration
• Web-scraper cost and currency, when applicable

Use the run history and the loader’s last error to diagnose failed or incomplete synchronization. After the loader succeeds, check its destination folder and then update any index that reads that folder.

Source deletion behavior

Some providers expose an option such as Delete Unavailable Documents. When enabled, a completed loader run removes documents previously created by that loader that are no longer present in the source. Leave it off when BotDojo should retain old documents even after they disappear upstream.

This comparison is loader-specific: BotDojo tracks the loader ID, source document ID, run ID, and synchronization checkpoint on the stored records. It does not delete unrelated documents that happen to share the destination folder.

For the built-in scraper’s source settings, continue to Web Scraping.