Add SemanticTextSplitter with dual local/global similarity scoring #40004

Submission checklist

• This is a feature request, not a bug report or usage question.
• I added a clear and descriptive title that summarizes the feature request.
• I used the GitHub search to find a similar feature request and didn't find it.
• I checked the LangChain documentation and API reference to see if this feature already exists.
• This is not related to the langchain-community package.

Package (Required)

• langchain
• langchain-openai
• langchain-anthropic
• langchain-classic
• langchain-core
• langchain-model-profiles
• langchain-tests
• langchain-text-splitters
• langchain-chroma
• langchain-deepseek
• langchain-exa
• langchain-fireworks
• langchain-groq
• langchain-huggingface
• langchain-mistralai
• langchain-nomic
• langchain-ollama
• langchain-openrouter
• langchain-perplexity
• langchain-qdrant
• langchain-xai
• Other / not sure / general

Feature Description

I would like to propose a new semantic text splitter for langchain-text-splitters that combines local sentence similarity with global current-chunk similarity when deciding whether a sentence should be merged into the current chunk.

The core scoring function is:

score = α × sim(new_sentence, previous_sentence)
      + (1 − α) × sim(new_sentence, current_chunk)

### Use Case


### Use Case

```markdown
This is primarily intended for RAG ingestion pipelines where semantic coherence within each retrieved chunk is important.

Character-based splitters are fast and predictable, but they do not directly consider semantic relationships between sentences. Purely adjacent semantic similarity can identify strong local boundaries, but it can also be affected by short-term similarity fluctuations or gradual topic drift.

For example, a sequence of sentences may temporarily have lower similarity while still discussing the same broader topic. Conversely, a sentence may be highly similar to the immediately preceding sentence while moving away from the overall topic of the current chunk.

The proposed dual-score approach addresses this by considering both:

- similarity to the previous sentence
- similarity to the semantic representation of the current chunk

This could be useful for:

- RAG document ingestion
- Long-form technical documentation
- Research papers
- Mixed-topic documents
- Knowledge bases
- Documents where preserving semantic units is more important than arbitrary character boundaries

The implementation also includes hard sentence/token constraints because semantic coherence alone does not guarantee that chunks remain suitable for embedding models, vector stores, rerankers, or LLM context windows.

### Proposed Solution

## Proposed Solution

I propose adding a semantic text splitter to `langchain-text-splitters` based on a dual local/global similarity scoring strategy.

The key idea is to evaluate a candidate sentence against both the immediately preceding sentence and the semantic representation of the current chunk.

### 1. Sentence-level Embeddings

First, split the document into sentences and generate embeddings for all sentences in a single batch operation:

```python
embeddings = model.encode(
    sentences,
    normalize_embeddings=True,
)

The resulting embeddings are cached and reused throughout the splitting process.

This avoids repeatedly embedding the same sentences during similarity calculations.

2. Adjacent Sentence Similarity

Calculate the similarity between every pair of adjacent sentences using the cached embeddings:

adjacent_similarity[i] = cosine(
    embeddings[i],
    embeddings[i + 1],
)

These similarities are used to identify potential semantic breakpoints and to derive a document-adaptive threshold.

3. Document-Adaptive Threshold

The initial implementation uses:

threshold = mean(adjacent_similarities) - std(adjacent_similarities)

This allows the threshold to adapt to the semantic cohesion of the current document rather than requiring a single hard-coded similarity value.

The implementation also supports alternative threshold strategies:

• std
• percentile
• gradient
• fixed

This could be extended or adjusted based on benchmarking results.

4. Dual Local/Global Similarity

For each new sentence, calculate two similarity values.

Local similarity

Similarity between the new sentence and the immediately preceding sentence:

local_similarity = cosine(
    new_embedding,
    previous_sentence_embedding,
)

This captures immediate semantic continuity.

Global similarity

Similarity between the new sentence and the semantic representation of the current chunk:

global_similarity = cosine(
    new_embedding,
    current_chunk_embedding,
)

This captures whether the sentence still belongs to the broader topic of the current chunk.

The final score is:

score = (
    alpha * local_similarity
    + (1 - alpha) * global_similarity
)

where alpha controls the relative importance of local versus global coherence.

5. Merge / Split Decision

If:

score >= threshold

the sentence is merged into the current chunk.

If:

score < threshold

a semantic boundary is considered, subject to the configured minimum chunk size.

Conceptually:

                 New sentence
                       |
             +---------+---------+
             |                   |
       Local similarity    Global similarity
             |                   |
             +---------+---------+
                       |
                  Dual score
                       |
                  Threshold
                       |
              +--------+--------+
              |                 |
            MERGE              SPLIT

6. Running Chunk Representation

Instead of repeatedly calculating the mean embedding of every sentence currently contained in the chunk, maintain an incremental chunk representation:

chunk_embedding = (
    chunk_embedding * N
    + new_embedding
) / (N + 1)

The resulting vector is normalized after the update:

chunk_embedding /= np.linalg.norm(chunk_embedding)

This keeps the chunk representation suitable for cosine similarity while avoiding repeated aggregation over all sentences in the chunk.

7. Chunk Size Constraints

Semantic coherence alone should not allow chunks to grow indefinitely.

The proposed implementation therefore combines semantic boundaries with optional hard constraints:

• min_sentences
• max_sentences
• max_tokens

A hard size constraint can force a boundary even when the semantic score remains above the threshold.

This is important for RAG systems where chunks must remain within practical embedding and LLM context limits.

8. Optional Sentence Overlap

The splitter can optionally preserve a configurable number of trailing sentences when creating a new chunk:

chunk_overlap_sentences=1

For example:

Chunk 1:
Sentence A
Sentence B
Sentence C
Sentence D

Chunk 2:
Sentence D
Sentence E
Sentence F
Sentence G

This can help preserve contextual continuity across retrieval boundaries.

9. Embedding Model Injection

Rather than requiring the splitter to instantiate its own embedding model, the implementation supports injecting a pre-loaded model.

For example:

model = SentenceTransformer("all-MiniLM-L6-v2")

splitter = SemanticTextSplitter(
    model=model,
    alpha=0.6,
)

This allows applications to reuse an embedding model across components and avoids unnecessary model initialization.

For LangChain integration, this could potentially be adapted to the existing LangChain Embeddings abstraction.

10. Proposed Pipeline

The complete process is:

Document
   |
   v
Sentence Tokenization
   |
   v
Batch Sentence Embeddings
   |
   v
Cached Embeddings
   |
   +----------------------+
   |                      |
   v                      v
Adjacent Similarity    Chunk Building
   |                      |
   v                      |
Breakpoint Threshold      |
   |                      |
   +----------+-----------+
              |
              v
       Dual Similarity
       /             \
      /               \
 Local Similarity   Global Similarity
      \               /
       \             /
        v           v
          Dual Score
              |
              v
        Merge / Split
              |
              v
       Size Constraints
              |
              v
        Semantic Chunks

11. Example

Given:

Machine learning is a subset of artificial intelligence.
It enables computers to learn from data without being explicitly programmed.
Supervised learning uses labeled datasets to train models.
Unsupervised learning finds hidden patterns in unlabeled data.
The Eiffel Tower is located in Paris, France.
It was built between 1887 and 1889 as the entrance arch for the 1889 World's Fair.
Python is a high-level, general-purpose programming language.
It emphasizes code readability with the use of significant indentation.

The semantic splitter should ideally produce chunks similar to:

Chunk 1:
Machine learning is a subset of artificial intelligence.
It enables computers to learn from data without being explicitly programmed.
Supervised learning uses labeled datasets to train models.
Unsupervised learning finds hidden patterns in unlabeled data.

Chunk 2:
The Eiffel Tower is located in Paris, France.
It was built between 1887 and 1889 as the entrance arch for the 1889 World's Fair.

Chunk 3:
Python is a high-level, general-purpose programming language.
It emphasizes code readability with the use of significant indentation.

The exact boundaries would depend on the embedding model and threshold strategy.

12. LangChain Integration

I would expect the final implementation to integrate with the existing TextSplitter abstraction and use LangChain's embedding interfaces rather than requiring a specific embedding provider.

I am not proposing the attached implementation as a drop-in PR at this stage.

The attached Python file is intended to demonstrate the algorithm and implementation details so that the maintainers can evaluate the approach first.

If the approach is considered useful, I would be happy to adapt it to LangChain's existing architecture, add unit tests and documentation, and benchmark it against existing splitters.

13. Evaluation

Before proposing a PR, I would like to evaluate the approach against existing LangChain splitters, particularly:

• RecursiveCharacterTextSplitter
• Existing semantic splitting functionality

Potential evaluation metrics include:

• Semantic coherence within chunks
• Accuracy of topic-boundary detection
• Average and median chunk size
• Number of generated chunks
• Token distribution
• Embedding calls and embedding cost
• Processing latency
• Retrieval Recall@K
• MRR
• Context precision
• Context recall
• End-to-end RAG answer quality

The goal would be to determine whether the additional global coherence signal provides a measurable improvement over existing semantic splitting approaches and whether that improvement justifies the additional computational cost.

Alternatives Considered

Alternatives Considered

I considered several existing and simpler approaches before arriving at the proposed dual local/global similarity strategy.

1. RecursiveCharacterTextSplitter

RecursiveCharacterTextSplitter is a strong general-purpose option because it is fast, deterministic, and provides explicit chunk-size and overlap controls.

However, it primarily makes splitting decisions based on separators and chunk size rather than semantic similarity.

For example, a topic transition can occur in the middle of an otherwise valid character window, causing semantically unrelated content to be placed in the same chunk.

I would continue to consider RecursiveCharacterTextSplitter a useful option when:

• predictable chunk sizes are the primary requirement
• embedding-based processing is too expensive
• deterministic and fast preprocessing is preferred

The proposed splitter is intended for cases where semantic coherence is more important than purely structural boundaries.

2. Existing Semantic Chunking

I also considered the existing semantic splitting functionality in LangChain.

Embedding-based semantic splitting is already a better fit than character-based splitting when the goal is to identify meaningful semantic boundaries.

The proposed approach differs primarily in how the decision is made.

Instead of relying only on the relationship between adjacent sentences, the proposed approach considers both:

Local:
sim(new_sentence, previous_sentence)

Global:
sim(new_sentence, current_chunk)

The two signals are combined as:

score = (
    alpha * local_similarity
    + (1 - alpha) * global_similarity
)

This is intended to make the splitting decision less dependent on a single adjacent sentence and provide an explicit signal for maintaining coherence with the overall current chunk.

3. Pure Adjacent-Sentence Similarity

A straightforward semantic splitter can use:

similarity = cosine(
    new_sentence_embedding,
    previous_sentence_embedding,
)

and create a boundary when similarity falls below a threshold.

This is simple and computationally attractive, but it only considers local continuity.

A potential limitation is that the immediately preceding sentence may not accurately represent the topic of the entire chunk.

For example:

Sentence A ── related ──> Sentence B
Sentence B ── related ──> Sentence C
Sentence C ── weakly related ──> Sentence D

Even if Sentence D remains relevant to the topic introduced by Sentence A, a low similarity with Sentence C could cause an unnecessary split.

The proposed global component attempts to provide additional context:

Sentence D
   |
   +--> similarity with Sentence C
   |
   +--> similarity with current chunk

4. Fixed Similarity Threshold

Another option is to use a manually selected threshold:

if similarity < 0.70:
    split()

The problem is that an appropriate threshold can vary significantly depending on:

• embedding model
• document domain
• writing style
• document structure
• language
• semantic density

The proposed implementation therefore supports a document-adaptive threshold strategy:

threshold = mean(adjacent_similarities) - std(adjacent_similarities)

It also supports alternative strategies such as percentile, gradient-based, and fixed thresholds so that the behavior can be evaluated empirically.

5. Recomputing the Chunk Embedding

A simpler implementation of global similarity could repeatedly calculate the mean embedding of all sentences currently in the chunk:

chunk_embedding = np.mean(
    embeddings[start:index],
    axis=0,
)

Although conceptually simple, repeatedly aggregating an increasingly large chunk introduces unnecessary computation.

The proposed implementation instead maintains a running chunk representation:

chunk_embedding = (
    chunk_embedding * N
    + new_embedding
) / (N + 1)

The resulting vector is normalized after each update.

This provides an incremental representation of the current chunk without repeatedly processing all previous sentence embeddings.

6. Re-embedding During Chunk Construction

Another possible implementation would embed sentences as they are processed and potentially re-embed accumulated chunks whenever a new semantic decision is required.

I avoided this because the same sentence embeddings can be reused for:

• adjacent similarity
• local similarity
• global similarity
• chunk representation updates

The proposed implementation therefore performs one batch embedding operation per document and reuses the resulting vectors.

7. Character or Token Window with Semantic Post-processing

Another possible strategy is to first create fixed-size chunks and then use embeddings to merge or split them.

This can provide strong token-size control, but it introduces a two-stage process where semantic boundaries may already have been lost during the initial fixed-size split.

The proposed approach instead makes semantic decisions at the sentence level while applying hard size constraints as safety limits.

8. Why the Proposed Approach

The goal is not to replace the existing splitters.

The trade-offs can be summarized as:

The proposed approach is therefore intended as an additional option for workloads where preserving semantic coherence within chunks is particularly important, especially RAG ingestion pipelines.

I would prefer to validate these trade-offs through benchmarks before suggesting that the proposed approach should replace or modify any existing LangChain splitter.

Additional Context

semantic_text_splitter.py

Social handles (optional)

No response