Chunk Text

The Chunk Text component divides large text content into smaller, manageable chunks based on token count. This component is essential for processing large documents in Retrieval Augmented Generation (RAG) systems by breaking them down into pieces that can be effectively embedded and retrieved from vector databases. By specifying chunk size and overlap parameters, users can optimize how their text is divided while maintaining context continuity between segments.

Overview

The Chunk Text component prevents context loss through intelligent overlap mechanisms while ensuring chunks fit within downstream processing limits. The text input typically comes from extracted document content via session variables, direct user input, or hardcoded values depending on the pipeline flow. Chunk size should be strategically chosen between 300-500 tokens for optimal balance between semantic coherence and processing efficiency - smaller chunks provide precise retrieval but may lose context, while larger chunks maintain context but may exceed model limits. The overlap size creates shared token boundaries between adjacent chunks, typically 20-50 tokens proportional to chunk size, preventing meaningful phrases or sentences from being split awkwardly and ensuring semantic continuity for better retrieval accuracy in RAG applications. When overlap is not needed for independent chunk processing, set overlap_size to 0. The component returns a simple list of text strings rather than complex objects, making it easy for downstream components to iterate through chunks in loops or process them individually.

Primary use cases include RAG pipeline preparation where chunks are embedded and stored in vector databases, large document processing that exceeds LLM context limits requiring segmentation, and text analysis workflows needing manageable content portions. The component automatically handles edge cases like text smaller than chunk size by returning single chunks without unnecessary splitting, and uses precise token counting rather than character counting for accurate LLM compatibility.

How to use:

Key Terms

When to Use

• Use when processing large documents or text that exceeds the context window of your language model.

• Ideal for preparing text for embedding into vector databases.

• Helpful when implementing RAG workflows that require manageable pieces of text for efficient retrieval.

• Use when you need to maintain semantic coherence across segments of a large document.

Component Configuration

Required Inputs

How It Works

1. The component first validates all input parameters to ensure they meet the requirements.

2. The text is then divided into chunks based on the specified chunk size.

3. Each chunk overlaps with the next by the specified overlap size to maintain context continuity.

4. If the text is smaller than the chunk size, the entire text is returned as a single chunk.

5. The component outputs an array of text chunks, each representing a portion of the original text.

Example Use Case

Configuration

• Text: "Artificial Intelligence (AI) is the simulation of human intelligence processes by machines, especially computer systems. These processes include learning (the acquisition of information and rules for using the information), reasoning (using the rules to reach approximate or definite conclusions), and self-correction. Particular applications of AI include expert systems, speech recognition, and machine vision. AI can be categorized as either weak or strong. Weak AI, also known as narrow AI, is an AI system designed to handle a specific task (such as facial recognition or internet searches) while strong AI, also known as artificial general intelligence, is an AI system with generalized human cognitive abilities."

• Chunk Size: 10

• Overlap Size: 5

Process

1. The component processes the provided text

2. It creates chunks with a size of 10 tokens each

3. Each chunk overlaps with the next by 5 tokens

4. The output is an array of text chunks

Output

Best Practices

• Choose an appropriate chunk size based on your embedding model's capabilities and the downstream LLM context window.

• Set a reasonable overlap size to maintain context continuity. A good rule of thumb is 10-20% of the chunk size.

Troubleshooting

Limitations and Considerations

• Context limitations - Even with overlap, some context might be lost between chunks, especially for complex documents with long-range dependencies.

• No natural language understanding - The chunking algorithm does not understand the content semantically, so it does not automatically preserve paragraphs, sections, or other logical structures.