Package sk.iway.iwcm.rag.indexing

Class SlidingWindowChunker

java.lang.Object

sk.iway.iwcm.rag.indexing.SlidingWindowChunker

@Component
public class SlidingWindowChunker
extends Object

Splits text into bounded, overlapping chunks, preferring sentence or paragraph boundaries.

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static final record	SlidingWindowChunker.Chunk	Stores a passage and its start offset in the normalized, trimmed source text.

Constructor Summary

Constructors

Constructor	Description
SlidingWindowChunker()

Method Summary

Modifier and Type	Method	Description
List<String>	chunk(String text)	Splits text using the configured approximate chunk size and overlap.
List<String>	chunk(String text, int chunkSize, int overlap)	Chooses sentence or paragraph boundaries nearest the requested size and overlap.
List<SlidingWindowChunker.Chunk>	chunkWithOffsets(String text)	Splits text using the configured size and overlap while retaining passage positions.
List<SlidingWindowChunker.Chunk>	chunkWithOffsets(String text, int chunkSize, int overlap)	Retains source offsets for the same bounded, overlapping passages returned by chunk(String, int, int).

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

SlidingWindowChunker

public SlidingWindowChunker()

Method Details

chunk

public List<String> chunk(String text)

Splits text using the configured approximate chunk size and overlap.

Parameters:

text - the full text to split

Returns:

list of text chunks

chunk

public List<String> chunk(String text,
 int chunkSize,
 int overlap)

Chooses sentence or paragraph boundaries nearest the requested size and overlap. Each chunk adds new content, even when overlap is large. Chunks may exceed the target by up to 50%, with a minimum cap of two UTF-16 units to preserve surrogate pairs. Longer units split at whitespace, or inside an oversized token when necessary.

Parameters:

text - the full text to split

chunkSize - target number of characters per chunk; nonpositive values disable splitting

overlap - target overlap in characters, rounded to whole sentences or paragraphs

Returns:

nonempty chunks with approximate sizes and no gaps in the source content

chunkWithOffsets

public List<SlidingWindowChunker.Chunk> chunkWithOffsets(String text)

Splits text using the configured size and overlap while retaining passage positions.

Parameters:

text - full source text

Returns:

chunks whose offsets refer to the source with normalized newlines and trimmed edges

chunkWithOffsets

public List<SlidingWindowChunker.Chunk> chunkWithOffsets(String text,
 int chunkSize,
 int overlap)

Retains source offsets for the same bounded, overlapping passages returned by chunk(String, int, int).

Parameters:

text - full source text

chunkSize - target number of characters per chunk

overlap - target overlap in characters

Returns:

chunks whose offsets refer to the source with normalized newlines and trimmed edges