Text Chunker for RAG (Preview & Tune Chunking)
Split long text into overlapping chunks for embedding and RAG pipelines. Preview recursive, paragraph, sentence, or fixed strategies with live token counts. Runs 100% in your browser.
Last updated:
Chunks
3
Total tokens (est.)
252
Avg tokens / chunk
84
Coverage
100%
Unique chars vs source
Retrieval-Augmented Generation (RAG) combines a language model with an external corpus. Instead of forcing the model to memorize everything, the retrieval step pulls the most relevant passages at query time.
ep pulls the most relevant passages at query time.Chunking is the step that decides how documents get split before they are embedded. Chunks that are too small lose context. Chunks that are too large dilute the embedding vector and cost more tokens per query. A good starting point for prose: - Chunk size: 300 to 800 tokens - Overlap: 10 to 20 percent of chunk size - Boundaries: prefer paragraph or sentence, avoid mid-word cuts
prefer paragraph or sentence, avoid mid-word cutsFor code, structured documents, and transcripts, custom splitters usually outperform generic ones. Always evaluate retrieval quality on a small labeled set — chunking parameters have a bigger impact on end-to-end quality than most teams realize.
Split long text into chunks by tokens or characters using recursive, paragraph, sentence, or fixed-size strategies. Configurable overlap. Perfect for preparing documents for RAG / embeddings pipelines. 100% local.
Tips & Best Practices
- ▸For RAG, chunks of 200-500 tokens with 10-20% overlap are a solid default. Too small and each chunk lacks context; too large and retrieval quality drops.
- ▸The 'recursive' strategy splits on paragraph → sentence → word → character in order, preserving structure where possible.
- ▸Overlap prevents information at chunk boundaries from being lost. Downstream deduplication is your friend.
Frequently Asked Questions
Which chunking strategy should I choose?
Recursive is the default in LangChain and works well for most prose — it tries paragraph, line, sentence, and word boundaries before hard-cutting. Use 'by paragraph' for well-structured documents (markdown, RFCs). 'By sentence' fits chatbot logs. 'Fixed' is only for exact-length experiments.
How large should chunks be for OpenAI embeddings?
text-embedding-3-small and 3-large both accept up to 8191 tokens per input. In practice, 300–800 token chunks strike the best balance between semantic granularity (higher recall) and context (less fragmentation). Overlap of ~10–20% of chunk size keeps concepts intact across boundaries.
Why is the tool in characters, not tokens?
Chunking by tokens requires running the tokenizer for every candidate cut — slow and imprecise since different models have different tokenizers. We split by characters (fast, deterministic) and show the estimated token count of each resulting chunk so you can tune chunk_size against your model's token budget.
Related Tools
Cosine Similarity
Compute cosine similarity, dot product, and Euclidean distance between two vectors online. Perfect for debugging LLM embeddings, semantic search, and RAG pipelines. 100% local — vectors never leave your browser.
AI Cost Calculator
Estimate monthly LLM API cost across GPT-4o, Claude 3.5, Gemini, DeepSeek and more. Compare providers side-by-side with prompt caching and batch discounts.
Token Counter
Estimate the number of tokens in your prompt for GPT-4, GPT-3.5, Claude, and Gemini. Predict API costs before you spend.
JSON Formatter
Format, beautify, and validate JSON online. Free, fast, and 100% local — your data never leaves your browser.
JSON Validator
Validate JSON online with instant error location and structure statistics (depth, keys, node types). 100% local — your JSON never leaves the browser.
JSON → TypeScript
Convert JSON to TypeScript interfaces instantly. Generate strongly-typed TS types from any JSON sample.