Data

Import organizational data, inspect stored documents, and manage the Data Loaders, indexes, and errors that keep it searchable for flows and agents.

5 min read

Data is the project-level home for organizational knowledge used by flows, agents, and workspaces. Start here when you want to import files, scrape a website, or synchronize records from a connected system such as Google Drive, Notion, or S3.

The Data area brings together four things you will manage:

Documents are the stored files and records organized in project folders.
Data Loaders import and synchronize documents from websites, BotDojo flows and repos, or connected systems.
Indexes turn selected documents into searchable parts and embeddings.
Errors and sync status show which loaders or indexed documents need attention.

Start by importing data

In a project, select Data from the left navigation. The route ends in /data/bucket and opens the Documents page in the Folders view. This is the main starting point for Data. Each top-level folder is a destination for related documents and summarizes its indexes, loaders, and record count.

The light-mode Data page with the Folders, Data Loaders, and Indexes views and the Import Data action

Select Import Data, then complete the three steps:

Create Folder — name the destination that will contain the imported documents.
Add Data — upload files or a ZIP archive, scrape a website, or choose a connected integration.
Create Index — choose how BotDojo creates embeddings and searchable document parts, then start the initial index update.

The light-mode Add Data workflow showing Create Folder, Add Data, and Create Index

The workflow always creates a destination folder and an index. When you import from a website or connected source, BotDojo also creates a Data Loader so you can run the import again or keep it synchronized on a schedule. Direct file and ZIP uploads are stored as documents without a loader.

Inspect documents, indexes, loaders, and errors

Select a folder from the Folders view to open its overview. This is the fastest place to understand whether imported data is available and searchable.

A light-mode folder overview with Browse Documents and Files, index status, Needs Attention, and Data Loaders

What to inspect	Where to find it	What it tells you
Documents	Select Browse Documents & Files.	View the stored files and records, paths, content types, metadata, and indexing state.
Indexes	Review the folder's Indexes card or open the project-wide Indexes view.	See the embedding model, indexed and total document counts, and whether the index is synchronized.
Data Loaders	Review the folder's Data Loaders card or open the project-wide Data Loaders view.	See which sources write to the folder, their schedules, run history, and latest status.
Errors and sync issues	Check Needs Attention on an index and the status shown for each loader.	Open the affected index or loader to inspect failed jobs, document errors, and configuration problems.

Errors are not a separate top-level view. Start from the affected folder, then open the loader run history or index jobs to see the failed operation, its log, and any document-level errors.

The project-wide view buttons are useful when you need to compare all loaders or indexes. The folder overview is more useful when you are troubleshooting one body of data because it shows the documents, loaders, and indexes together.

How the pieces fit together

Diagram showing how Data Loaders write documents to project folders and indexes make those documents searchable

A document is the source of truth. It has content, a logical path, a content type, dates, a version, and JSON metadata. Documents can be uploaded by a person, written by a flow or workspace, sent through the API, or created by a Data Loader.

A Data Loader is a repeatable import. It reads a website, BotDojo flow history, a BotDojo-managed repo, or a connected system, then writes the resulting documents to one destination folder. A loader can run manually, on a schedule, or—when supported—after a source event such as a repo commit. Its run history shows what happened during each synchronization.

An index reads documents from a folder. It splits each document into smaller document parts, copies document metadata to those parts, and creates an embedding for semantic search. Flows and agents query the index; they do not query the loader.

This separation means:

Removing or changing an index does not remove the original documents.
One folder can feed more than one index, which is useful when comparing embedding models or chunking strategies.
A loader can refresh its destination folder without changing how an index is configured.
A document can appear in multiple indexes.

Follow a document through Data

Content enters the project through Import Data, a flow, a workspace, or the API.
BotDojo stores it as a document at a logical path inside a project folder.
An index selects the documents under its configured Data Location.
The index chunks each selected document and creates embeddings for the resulting parts.
A flow or agent searches the index and receives matching text, metadata, references, and relevance scores.

When a source changes, run the Data Loader again to update the documents. If Auto Index when documents are added or modified is enabled, BotDojo schedules the index work automatically. Otherwise, open the index and select Run Index Update.

Choose what to do next

Open Documents to inspect stored content and metadata.
Use Data Loaders to configure synchronized imports and review runs.
Follow Create an Index to configure semantic retrieval.
Use the Data API when an application or pipeline should write documents directly.