Get started
Introduction
Semantic Chunkers is a library for splitting text into meaningful chunks based on semantic similarity. Rather than splitting on fixed character counts or simple delimiters, it uses embedding models to identify natural breakpoints in your text.
What is Semantic Chunkers?
Semantic Chunkers enables:
- Meaningful splits: Break text at natural semantic boundaries rather than arbitrary positions
- Better retrieval: Chunks that preserve context lead to more accurate search and RAG pipelines
- Flexible chunking: Multiple chunking strategies to suit different content types
- Easy integration: Works with popular embedding providers like OpenAI, Cohere, and local models
Key Features
- Statistical chunking: Split based on statistical analysis of embedding similarities
- Consecutive chunking: Group consecutive sentences that are semantically similar
- Cumulative chunking: Build chunks by accumulating sentences until a semantic shift
- Multiple encoders: Support for OpenAI, Cohere, Hugging Face, and more
Getting Started
For a quick introduction to using Semantic Chunkers, check out our quickstart guide.