Package sk.iway.iwcm.rag.indexing

Class MarkdownContentExtractor

java.lang.Object

sk.iway.iwcm.rag.indexing.MarkdownContentExtractor

All Implemented Interfaces:

ContentExtractor<String>

@Component
public class MarkdownContentExtractor
extends Object
implements ContentExtractor<String>

Prepares Markdown for chunking while preserving document structure and code examples.

Constructor Summary

Constructors

Constructor	Description
MarkdownContentExtractor()

Method Summary

Modifier and Type	Method	Description
List<String>	addHeadingContext(String text, String title, List<SlidingWindowChunker.Chunk> chunks)	Adds the document title and the shared heading hierarchy of each passage to its embedding input.
String	extractText(String markdown)	Removes front matter, link destinations, image paths, and hidden HTML before chunking.
String	extractTitle(String markdown, String sourcePath)	Uses the first level-one heading outside comments and code, falling back to the relative file path.
RagEntityType	getEntityType()	Returns the entity type identifier (e.g., "document").

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

MarkdownContentExtractor

public MarkdownContentExtractor()

Method Details

extractText

public String extractText(String markdown)

Removes front matter, link destinations, image paths, and hidden HTML before chunking. Link labels, existing image descriptions, headings, lists, tables, and code examples are retained.

Specified by:

extractText in interface ContentExtractor<String>

Parameters:

markdown - source Markdown, or null

Returns:

cleaned Markdown, or an empty string for null input

extractTitle

public String extractTitle(String markdown,
 String sourcePath)

Uses the first level-one heading outside comments and code, falling back to the relative file path.

Parameters:

markdown - source Markdown, or null

sourcePath - non-null relative path used when no heading is found

Returns:

heading or fallback path, truncated to at most 512 characters

addHeadingContext

public List<String> addHeadingContext(String text,
 String title,
 List<SlidingWindowChunker.Chunk> chunks)

Adds the document title and the shared heading hierarchy of each passage to its embedding input. Merged sibling subsections use their common parent. Passage bodies remain unchanged, code headings are ignored, and overlapping passages retain context at their own source positions.

Parameters:

text - cleaned Markdown with normalized newlines, as supplied to the chunker

title - document title or source-path fallback

chunks - ordered chunks with offsets in the trimmed source text

Returns:

contextual embedding inputs in chunk order

getEntityType

public RagEntityType getEntityType()

Description copied from interface: ContentExtractor

Returns the entity type identifier (e.g., "document").

Specified by:

getEntityType in interface ContentExtractor<String>