Node Parser Modules

File-Based Node Parsers

There are several file-based node parsers, that will create nodes based on the type of content that is being parsed (JSON, Markdown, etc.)

The simplest flow is to combine the FlatFileReader with the SimpleFileNodeParser to automatically use the best node parser for each type of content. Then, you may want to chain the file-based node parser with a text-based node parser to account for the actual length of the text.

SimpleFileNodeParser

from llama_index.node_parser.file import SimpleFileNodeParser
from llama_index.readers.file.flat_reader import FlatReader

md_docs = FlatReader().load_data("./test.md")

parser = SimpleFileNodeParser()
md_nodes = parser.get_nodes_from_documents(md_docs)

HTMLNodeParser

This node parser uses beautifulsoup to parse raw HTML.

By default, it will parse a select subset of HTML tags, but you can override this.

The default tags are: ["p", "h1", "h2", "h3", "h4", "h5", "h6", "li", "b", "i", "u", "section"]

from llama_index.node_parser import HTMLNodeParser

parser = HTMLNodeParser(tags=["p", "h1"])  # optional list of tags
nodes = parser.get_nodes_from_documents(html_docs)

JSONNodeParser

The JSONNodeParser parses raw JSON.

from llama_index import JSONNodeParser

parser = JSONNodeParser()

nodes = parser.get_nodes_from_documents(json_docs)

MarkdownNodeParser

The MarkdownNodeParser parses raw markdown text.

from llama_index import MarkdownNodeParser

parser = MarkdownNodeParser()

nodes = parser.get_nodes_from_documents(markdown_docs)

Text-Splitters

CodeSplitter

Splits raw code-text based on the language it is written in.

Check the full list of supported languages here.

from llama_index.text_splitter import CodeSplitter

splitter = CodeSplitter(
    language="python",
    chunk_lines=40,  # lines per chunk
    chunk_lines_overlap=15,  # lines overlap between chunks
    max_chars=1500,  # max chars per chunk
)
nodes = splitter.get_nodes_from_documents(documents)

LangchainNodeParser

You can also wrap any existing text splitter from langchain with a node parser.

from langchain.text_splitter import RecursiveCharacterTextSplitter
from llama_index.node_parser import LangchainNodeParser

parser = LangchainNodeParser(RecursiveCharacterTextSplitter())
nodes = parser.get_nodes_from_documents(documents)

SentenceSplitter

The SentenceSplitter attempts to split text while respecting the boundaries of sentences.

from llama_index.text_splitter import SentenceSplitter

splitter = SentenceSplitter(
    chunk_size=1024,
    chunk_overlap=20,
)
nodes = splitter.get_nodes_from_documents(documents)

SentenceWindowNodeParser

The SentenceWindowNodeParser is similar to other node parsers, except that it splits all documents into individual sentences. The resulting nodes also contain the surrounding “window” of sentences around each node in the metadata. Note that this metadata will not be visible to the LLM or embedding model.

This is most useful for generating embeddings that have a very specific scope. Then, combined with a MetadataReplacementNodePostProcessor, you can replace the sentence with it’s surrounding context before sending the node to the LLM.

An example of setting up the parser with default settings is below. In practice, you would usually only want to adjust the window size of sentences.

import nltk
from llama_index.node_parser import SentenceWindowNodeParser

node_parser = SentenceWindowNodeParser.from_defaults(
    # how many sentences on either side to capture
    window_size=3,
    # the metadata key that holds the window of surrounding sentences
    window_metadata_key="window",
    # the metadata key that holds the original sentence
    original_text_metadata_key="original_sentence",
)

A full example can be found here in combination with the MetadataReplacementNodePostProcessor.

TokenTextSplitter

The TokenTextSplitter attempts to split text while respecting the boundaries of sentences.

from llama_index.text_splitter import TokenTextSplitter

splitter = TokenTextSplitter(
    chunk_size=1024,
    chunk_overlap=20,
    separator=" ",
)
nodes = splitter.get_nodes_from_documents(documents)

Relation-Based Node Parsers

HierarchicalNodeParser

This node parser will chunk nodes into hierarchical nodes. This means a single input will be chunked into several hierarchies of chunk sizes, with each node containing a reference to it’s parent node.

When combined with the AutoMergingRetriever, this enables us to automatically replace retrieved nodes with their parents when a majority of children are retrieved. This process provides the LLM with more complete context for response synthesis.

from llama_index.node_parser import HierarchicalNodeParser

node_parser = HierarchicalNodeParser.from_defaults(
    chunk_sizes=[2048, 512, 128]
)

A full example can be found here in combination with the AutoMergingRetriever.