SemanticSplitterNodeParser

pydantic model llama_index.node_parser.SemanticSplitterNodeParser

Semantic node parser.

Splits a document into Nodes, with each node being a group of semantically related sentences.

Parameters

• buffer_size (int) – number of sentences to group together when evaluating semantic similarity

• embed_model – (BaseEmbedding): embedding model to use

• sentence_splitter (Optional[Callable]) – splits text into sentences

• include_metadata (bool) – whether to include metadata in nodes

• include_prev_next_rel (bool) – whether to include prev/next relationships

Show JSON schema

{
   "title": "SemanticSplitterNodeParser",
   "description": "Semantic node parser.\n\nSplits a document into Nodes, with each node being a group of semantically related sentences.\n\nArgs:\n    buffer_size (int): number of sentences to group together when evaluating semantic similarity\n    embed_model: (BaseEmbedding): embedding model to use\n    sentence_splitter (Optional[Callable]): splits text into sentences\n    include_metadata (bool): whether to include metadata in nodes\n    include_prev_next_rel (bool): whether to include prev/next relationships",
   "type": "object",
   "properties": {
      "include_metadata": {
         "title": "Include Metadata",
         "description": "Whether or not to consider metadata when splitting.",
         "default": true,
         "type": "boolean"
      },
      "include_prev_next_rel": {
         "title": "Include Prev Next Rel",
         "description": "Include prev/next node relationships.",
         "default": true,
         "type": "boolean"
      },
      "callback_manager": {
         "title": "Callback Manager"
      },
      "id_func": {
         "title": "Id Func"
      },
      "embed_model": {
         "title": "Embed Model"
      },
      "buffer_size": {
         "title": "Buffer Size",
         "description": "The number of sentences to group together when evaluating semantic similarity. Set to 1 to consider each sentence individually. Set to >1 to group sentences together.",
         "default": 1,
         "type": "integer"
      },
      "breakpoint_percentile_threshold": {
         "title": "Breakpoint Percentile Threshold",
         "description": "The percentile of cosine dissimilarity that must be exceeded between a group of sentences and the next to form a node.  The smaller this number is, the more nodes will be generated",
         "default": 95,
         "type": "integer"
      },
      "class_name": {
         "title": "Class Name",
         "type": "string",
         "default": "SemanticSplitterNodeParser"
      }
   }
}

Config

• arbitrary_types_allowed: bool = True

Fields

• buffer_size (int)

• embed_model (llama_index.core.embeddings.base.BaseEmbedding)

• sentence_splitter (Callable[[str], List[str]])

field buffer_size: int = 1

The number of sentences to group together when evaluating semantic similarity. Set to 1 to consider each sentence individually. Set to >1 to group sentences together.

field embed_model: BaseEmbedding [Required]

The embedding model to use to for semantic comparison

field sentence_splitter: Callable[[str], List[str]] [Optional]

The text splitter to use when splitting documents.

build_semantic_nodes_from_documents(documents: Sequence[Document], show_progress: bool = False) → List[BaseNode]

Build window nodes from documents.

classmethod class_name() → str

Get the class name, used as a unique ID in serialization.

This provides a key that makes serialization robust against actual class name changes.

classmethod from_defaults(embed_model: Optional[BaseEmbedding] = None, breakpoint_percentile_threshold: Optional[int] = 95, buffer_size: Optional[int] = 1, sentence_splitter: Optional[Callable[[str], List[str]]] = None, original_text_metadata_key: str = 'original_text', include_metadata: bool = True, include_prev_next_rel: bool = True, callback_manager: Optional[CallbackManager] = None, id_func: Optional[Callable[[int, Document], str]] = None) → SemanticSplitterNodeParser