Chunking

An index does not embed an entire document as one value. It splits the document into ordered parts, copies document metadata to each part, and embeds each part separately. Search returns the most relevant parts.

Current chunking strategies

Strategy	How it works	Use it when
By Character	Uses a recursive text splitter measured in characters.	You want a predictable default that works across ordinary text and Markdown.
By Token	Splits using model-style tokens rather than characters.	You need chunk limits to line up more closely with language-model context and cost.
Custom	Runs each document through a flow exported as a BotDojo chunker.	You need domain-specific sections, rewriting, summaries, translations, or custom part metadata.

Shared settings

Setting	What it does
Max Chunks per Document	Caps the number of parts generated from one document. Leave blank for no limit. A cap can control unusually large documents but may omit content after the limit.
Chunk Size	Target maximum size of each part in characters or tokens, depending on the strategy.
Chunk Overlap	Repeats content between adjacent parts so a fact near a boundary is not separated from its context.

The current form starts character and token strategies at chunk size 2000 and overlap 20.

Larger chunks preserve more surrounding context but can make a retrieved result less focused. Smaller chunks improve precision but may separate a statement from its qualifications, headings, or definitions. Overlap helps, but excessive overlap increases duplicate results and storage.

Custom chunker settings

Selecting Custom reveals:

• Custom Chunker: a flow exported with the BotDojo chunker package category.
• Provider-defined property overrides, when the selected chunker exposes them.
• Concurrency Level: how many chunker-flow executions run in parallel, from 1 to 20. The default is 10.

A custom chunker receives the document context and returns the parts that BotDojo stores and embeds. It can:

• Preserve headings, sections, or table boundaries.
• Add part-level metadata such as section_title, jurisdiction, or product.
• Summarize or rewrite a part to add document-level context.
• Normalize multilingual content.

Custom chunking is more flexible and usually more expensive than a built-in splitter. Keep the flow deterministic, preserve source references, and test a representative sample before processing a large folder.

Tune chunking with evidence

1. Create an index with a reasonable baseline.
2. Open Test Index and search for questions with known answers.
3. Inspect whether the returned part contains the answer and enough context to use it correctly.
4. Check for duplicated, truncated, or unrelated results.
5. Create a second index on the same folder with a different chunking configuration.
6. Compare retrieval quality before replacing the index used by a production flow.

Because the original documents remain in the folder, multiple indexes can safely test different chunking and embedding choices over the same source data.