Indexes
Create fast semantic search across organizational data for RAG, deep research, and agent workflows.
Indexes give agents fast semantic search across organizational knowledge such as Google Drive files, call transcripts, support tickets, and internal documentation. They power retrieval-augmented generation (RAG) and deep research by finding relevant passages by meaning while preserving metadata for filtering and citations.
An agent can browse, open, and work with Google Drive directly through a connection without an index. Create an index when the agent needs to search quickly across many files or sources, run repeated research queries, or combine semantic retrieval with exact metadata filters.
A connection provides live access to the source and its actions. An index provides an optimized, searchable copy. It reads documents from a configured Data Location, splits them into document parts, creates embeddings, and returns relevant text and metadata when a flow or agent searches it.
The original documents remain in their folders. Rebuilding, changing, or deleting an index changes the searchable copy without deleting those source documents.
Documents, records, and metadata
An integration-backed Data Loader stores each source record as a BotDojo document. For example, a record can be a Notion page, Jira issue, Zendesk ticket, Slack thread, or Google Drive file.
Each document contains:
- readable content that describes the record and can be split into searchable parts;
- a
metaJSON object containing attributes and related information such as source IDs, URLs, status, owner, timestamps, tags, project, channel, or record type.
Some loaders write provider attributes directly to the document metadata. Others render a Markdown document and place selected attributes in YAML front matter. During indexing, BotDojo reads those attributes and copies the combined document metadata to every searchable part.
The content answers “what is this record about?” Metadata answers “which record is it, where did it come from, and which exact attributes does it have?”
Agents and flows query the index rather than the Data Loader. A query returns matching part text together with its metadata, references, document identifiers, and relevance score. The original document remains the source of truth in its project folder.
Combine semantic search with metadata filters
BotDojo combines two retrieval methods in one index query:
- Semantic search embeds the query and compares its meaning with the embeddings of document parts.
- Metadata filtering applies exact conditions such as status, region, owner, date, record type, or source ID.
Semantic search can find a support ticket that describes “the customer cannot reset a password” even when those exact words are not present. A metadata filter can restrict the same search to open tickets for the enterprise product.
Start with semantic search to verify the content is retrievable. Then add the smallest metadata filter that represents the records the agent is allowed or expected to use. The same Query panel also provides Metadata Query for inspecting stored rows and validating filters without an embedding search.
Optimize metadata filters at scale
JSON metadata is flexible, but extracting and converting values from meta for every candidate row can become slow on a large index or a frequently used query.
Use these optimization layers:
| Layer | Use it when | What it does |
|---|---|---|
| JSON metadata | Fields are exploratory, infrequently queried, or vary between documents. | Keeps the complete source attributes in the flexible meta object. |
| Materialized columns | The same fields are filtered, sorted, compared, or selected repeatedly. | Extracts metadata values into typed stored columns such as col_status, col_priority, or col_created_at. |
| Column indexes | A large index repeatedly filters or sorts on the same materialized column or column combination. | Adds a B-tree index that accelerates those access patterns. |
Materialized columns do not replace or change the original metadata. They provide a fast, typed projection of selected values. Column indexes are an additional optimization over those projected columns; add them deliberately because they use storage and add work to index updates.
For the configuration workflow, expression examples, data types, query syntax, and rebuild behavior, see Materialized Columns.
Choose what to do
- Create an Index to select documents, embeddings, chunking, and index storage.
- Manage an Index to query semantic and metadata results, inspect index jobs, monitor health, and run updates.
- Chunking to choose how documents are divided into searchable parts.
- Metadata and Markdown to understand source metadata, front matter, filters, and citations.
- Materialized Columns to create typed metadata columns and column indexes for fast filtering at scale.
Keep an index current
Enable Auto Index when documents are added or modified when new or changed documents should schedule index work automatically. Otherwise, open the index and select Run Index Update after documents change.
Use the health counts on the index page to identify documents that need embeddings, are out of sync, or have errors. Select Query to verify semantic retrieval and materialized metadata, and use Index Jobs to monitor updates or diagnose failures.