LangChain Text Splitters: Optimizing LLM Input for Efficiency and Accuracy

Our previous article covered LangChain's document loaders. However, LLMs have context window size limitations (measured in tokens). Exceeding this limit truncates data, compromising accuracy and increasing costs. The solution? Send only relevant data to the LLM, requiring data splitting. Enter LangChain's Text Splitters.

Key Concepts:

1. The Crucial Role of Text Splitters: Understand why efficient text splitting is vital for optimizing LLM applications, balancing context window size and cost.
2. Diverse Text Splitting Techniques: Explore various methods, including character counts, token counts, recursive splitting, and techniques tailored to HTML, code, and JSON structures.
3. LangChain Text Splitter Implementation: Learn practical application, including installation, code examples for text splitting, and handling diverse data formats.
4. Semantic Splitting for Enhanced Relevance: Discover how sentence embeddings and cosine similarity create semantically coherent chunks, maximizing relevance.

Table of Contents:

• What are Text Splitters?
• Data Splitting Methods
• Character Count-Based Splitting
• Recursive Splitting
• Token Count-Based Splitting
• Handling HTML
• Code-Specific Splitting
• JSON Data Handling
• Semantic Chunking
• Frequently Asked Questions

What are Text Splitters?

Text splitters divide large text into smaller, manageable chunks for improved LLM query relevance. They work directly on raw text or LangChain document objects. Multiple methods cater to different content types and use cases.

Data Splitting Methods

LangChain Text Splitters are crucial for efficient large document processing. They improve performance, contextual understanding, enable parallel processing, and facilitate better data management. Let's examine several methods:

Prerequisites: Install the package using pip install langchain_text_splitters

Character Count-Based Splitting

This method splits text based on character count, using a specified separator.

from langchain_community.document_loaders import UnstructuredPDFLoader
from langchain_text_splitters import CharacterTextSplitter

# Load data (replace with your PDF path)
loader = UnstructuredPDFLoader('how-to-formulate-successful-business-strategy.pdf', mode='single')
data = loader.load()

text_splitter = CharacterTextSplitter(separator="\n", chunk_size=500, chunk_overlap=0, is_separator_regex=False)
texts = text_splitter.split_documents(data)
len(texts) # Output: Number of chunks

This example splits text into 500-character chunks, using newline characters as separators.

Recursive Splitting

This uses multiple separators sequentially until chunks are below chunk_size. Useful for sentence-level splitting.

from langchain_text_splitters import RecursiveCharacterTextSplitter

recursive_splitter = RecursiveCharacterTextSplitter(separators=["\n\n", "\n", r"(?>> 293

# ... (rest of the code remains similar)

Token Count-Based Splitting

LLMs use tokens; splitting by token count is more accurate. This example uses the o200k_base encoding (check the GitHub link for model/encoding mappings).

from langchain_text_splitters import TokenTextSplitter

text_splitter = TokenTextSplitter(encoding_name='o200k_base', chunk_size=50, chunk_overlap=0)
texts = text_splitter.split_documents(data)
len(texts) # Output: Number of chunks

Recursive splitting can also be combined with token counting.

For plain text, recursive splitting with character or token counting is generally preferred.

Handling HTML

For structured data like HTML, splitting should respect the structure. This example splits based on HTML headers.

from langchain_text_splitters import HTMLHeaderTextSplitter

headers_to_split_on = [("h1", "Header 1"), ("h2", "Header 2"), ("h3", "Header 3")]
html_splitter = HTMLHeaderTextSplitter(headers_to_split_on, return_each_element=True)
html_header_splits = html_splitter.split_text_from_url('https://diataxis.fr/')
len(html_header_splits) # Output: Number of chunks

HTMLSectionSplitter allows splitting based on other sections.

Code-Specific Splitting

Programming languages have unique structures. This example uses syntax-aware splitting for Python code.

from langchain_text_splitters import RecursiveCharacterTextSplitter, Language

# ... (Python code example) ...

python_splitter = RecursiveCharacterTextSplitter.from_language(language=Language.PYTHON, chunk_size=100, chunk_overlap=0)
python_docs = python_splitter.create_documents([PYTHON_CODE])

JSON Data Handling

Nested JSON objects can be split while preserving key relationships.

from langchain_text_splitters import RecursiveJsonSplitter

# ... (JSON data example) ...

splitter = RecursiveJsonSplitter(max_chunk_size=200, min_chunk_size=20)
chunks = splitter.split_text(json_data, convert_lists=True)

Semantic Chunking

This method uses sentence embeddings and cosine similarity to group semantically related sentences.

from langchain_experimental.text_splitter import SemanticChunker
from langchain_openai.embeddings import OpenAIEmbeddings # Requires OpenAI API key

# ... (code using OpenAIEmbeddings and SemanticChunker) ...

Conclusion

LangChain offers various text splitting methods, each suited for different data types. Choosing the right method optimizes LLM input, improving accuracy and reducing costs.

Frequently Asked Questions

(Q&A section remains largely the same, with minor wording adjustments for clarity and flow.)

The above is the detailed content of 7 Ways to Split Data Using LangChain Text Splitters - Analytics Vidhya. For more information, please follow other related articles on the PHP Chinese website!