Metadata and Markdown

Metadata travels with a document into every index part created from it. It is the link between stored content, retrieval filters, and citations.

From source record to searchable document

An integration-backed Data Loader stores each source record as a BotDojo document. The document’s readable content describes the record; its JSON metadata in meta holds attributes and related information about that record.

For example, a support record might use the ticket conversation as content and store these values as metadata:

{
  "source_id": "ticket_8472",
  "record_type": "support_ticket",
  "status": "open",
  "priority": 2,
  "region": "north_america",
  "reference_url": "https://support.example.com/tickets/8472"
}

The exact fields depend on the integration. Common values include a stable source ID, provider URL, title, owner, status, tags, timestamps, project, channel, locale, or other provider-specific attributes.

Some Data Loaders write attributes directly into document metadata. Others generate Markdown and put selected fields in YAML front matter. When BotDojo indexes the document, it combines the available metadata with system document fields and copies it to each searchable part.

An agent can therefore search the part text by meaning and filter the same query by exact metadata values. The result includes the matching text and the metadata needed to identify, cite, or act on the original record.

System document metadata

BotDojo tracks document fields such as:

Field	Meaning
path	Full logical path, for example /policies/returns.md.
folder	Logical folder that contains the document.
filename	File name at the end of the path.
size_bytes	Stored file size.
content_type	MIME type such as text/markdown.
created_date	Document creation date.
modified_date	Most recent document modification date.

Loader-provided fields can include a source record ID, URL, folder, locale, tags, or other provider-specific attributes.

Open a document and select Metadata to inspect or edit its JSON object. Select Save to persist the new value. Run any affected index again so its parts receive the updated metadata.

Citation fields

Use these fields when a flow should present sources:

• reference_url: canonical link to the source.
• reference_title or title: readable source label.

Retrieval returns part metadata and references alongside the matched text. A flow can pass those values to an agent or source display.

Markdown front matter

Markdown can keep custom metadata with the content by placing YAML front matter at the start of the file:

---
title: Return policy
department: Support
region: North America
reference_url: https://example.com/policies/returns
---

# Return policy

Customers may return an unopened item within 30 days.

Front matter is useful when the document should remain portable outside BotDojo. Loader metadata is often better when values come from the external source and should not be written into the content.

During Markdown indexing, BotDojo removes the front matter from the text that is embedded and makes its attributes available as metadata. This keeps configuration values out of the searchable prose while preserving them for filters and citations.

Filter semantic search

Select Query → Semantic Search. BotDojo embeds the search text to find parts with similar meaning, then applies the optional metadata filter. Use the WHERE expression or switch to JSON to narrow results. Start without a filter to verify retrieval, then add the smallest condition that represents the intended audience or content class.

Good filter candidates are stable, consistently populated fields such as department, region, language, product, or record type. Avoid filtering on display text that changes frequently.

For example, semantic search can find records about refund requests while the filter restricts results to open support records:

meta->>'record_type' = 'support_ticket'
AND meta->>'status' = 'open'

Metadata design guidelines

• Use consistent field names and value types across every document in an index.
• Preserve a stable source identifier so loader reruns update the correct document.
• Store canonical citation URLs rather than temporary signed URLs.
• Prefer a small set of purposeful fields over copying an entire source record.
• Treat metadata as project data: do not place secrets or credentials in it.

For frequently queried fields and large-index performance, continue to Materialized Columns.