Class RecursiveTextChunker

Recursive text chunking strategy.

Splits text using a hierarchy of separators (paragraphs, sentences, words) to maintain semantic coherence while respecting size constraints.

The algorithm tries to split on:

  1. Double newlines (paragraphs)
  2. Single newlines
  3. Sentences (. ! ?)
  4. Words (spaces)

Example

const chunker = new RecursiveTextChunker({
  chunkSize: 1000,
  chunkOverlap: 200,
  minChunkSize: 100
});

const result = await chunker.chunk({
  id: 'doc1',
  content: largeText,
  metadata: { source: 'manual.pdf' }
});

Implements

  • DocumentChunker

Constructors

constructor

Methods

chunk chunkMany estimateChunks

Properties

config

Constructors

constructor

Methods

chunk

  • chunk(document: Document): Promise<ChunkResult>

    Split a document into chunks

    Parameters

    • document: Document

      Document to chunk

    Returns Promise<ChunkResult>

    Chunked documents with metadata

chunkMany

  • chunkMany(documents: Document[]): Promise<ChunkResult[]>

    Split multiple documents into chunks

    Parameters

    • documents: Document[]

      Documents to chunk

    Returns Promise<ChunkResult[]>

    Array of chunk results

estimateChunks

  • estimateChunks(document: Document): number

    Estimate the number of chunks that will be created

    Parameters

    • document: Document

      Document to estimate

    Returns number

    Estimated number of chunks

Properties

Readonlyconfig

config: ChunkerConfig

Configuration for this chunker

Settings

Member Visibility

On This Page

Constructors
Methods
Properties