Text Chunker for RAG (Preview & Tune Chunking)
Split long text into overlapping chunks for embedding and RAG pipelines. Preview recursive, paragraph, sentence, or fixed strategies with live token counts. Runs 100% in your browser.
Last updated:
CommentsSplit long text into chunks by tokens or characters using recursive, paragraph, sentence, or fixed-size strategies. Configurable overlap. Perfect for preparing documents for RAG / embeddings pipelines. 100% local.
Chunks
3
Total tokens (est.)
252
Avg tokens / chunk
84
Coverage
100%
Unique chars vs source
Retrieval-Augmented Generation (RAG) combines a language model with an external corpus. Instead of forcing the model to memorize everything, the retrieval step pulls the most relevant passages at query time.
ep pulls the most relevant passages at query time.Chunking is the step that decides how documents get split before they are embedded. Chunks that are too small lose context. Chunks that are too large dilute the embedding vector and cost more tokens per query. A good starting point for prose: - Chunk size: 300 to 800 tokens - Overlap: 10 to 20 percent of chunk size - Boundaries: prefer paragraph or sentence, avoid mid-word cuts
prefer paragraph or sentence, avoid mid-word cutsFor code, structured documents, and transcripts, custom splitters usually outperform generic ones. Always evaluate retrieval quality on a small labeled set — chunking parameters have a bigger impact on end-to-end quality than most teams realize.
Tips & Best Practices
- ▸For RAG, chunks of 200-500 tokens with 10-20% overlap are a solid default. Too small and each chunk lacks context; too large and retrieval quality drops.
- ▸The 'recursive' strategy splits on paragraph → sentence → word → character in order, preserving structure where possible.
- ▸Overlap prevents information at chunk boundaries from being lost. Downstream deduplication is your friend.
Frequently Asked Questions
Which chunking strategy should I choose?
Recursive is the default in LangChain and works well for most prose — it tries paragraph, line, sentence, and word boundaries before hard-cutting. Use 'by paragraph' for well-structured documents (markdown, RFCs). 'By sentence' fits chatbot logs. 'Fixed' is only for exact-length experiments.
How large should chunks be for OpenAI embeddings?
text-embedding-3-small and 3-large both accept up to 8191 tokens per input. In practice, 300–800 token chunks strike the best balance between semantic granularity (higher recall) and context (less fragmentation). Overlap of ~10–20% of chunk size keeps concepts intact across boundaries.
Why is the tool in characters, not tokens?
Chunking by tokens requires running the tokenizer for every candidate cut — slow and imprecise since different models have different tokenizers. We split by characters (fast, deterministic) and show the estimated token count of each resulting chunk so you can tune chunk_size against your model's token budget.
Try Next
Token Counter
Count tokens in your prompt for GPT-4, GPT-3.5, Claude, and Gemini and predict API cost before you spend. Free, no signup — all counting runs locally, your prompt stays in your browser.
Related Tools
Vector Storage Calculator
Estimate how much memory or disk your embeddings need — enter vector count, dimensions, and precision (float32/float16/int8/binary) to see raw and indexed size. 100% local.
Prompt Cache Calculator
See how much prompt caching saves you. Enter your cached prefix size, per-request tokens, volume, and hit rate to compare cost with and without caching across models that support it. 100% local.
Batch API Cost Calculator
Compare real-time vs Batch API cost for a bulk job. Enter request count and per-request tokens to see how much the ~50% batch discount saves across models that support it. 100% local.
Prompt Injection Detector
Scan untrusted user input for common prompt-injection and jailbreak patterns — instruction overrides, jailbreak personas, system-prompt exfiltration — with a risk score. Heuristic, 100% local.
Cosine Similarity
Compute cosine similarity, dot product, and Euclidean distance between two vectors online. Perfect for debugging LLM embeddings, semantic search, and RAG pipelines. 100% local — vectors never leave your browser.
AI Cost Calculator
Estimate monthly LLM API cost across GPT-4o, Claude 3.5, Gemini, DeepSeek and 100+ models. Compare providers side-by-side with prompt-caching and batch discounts. Free, no signup — prices refreshed daily from the public LiteLLM catalog.