如何拆分 HTML

将 HTML 文档分割成易于管理的小块对于各种文本处理任务至关重要，例如自然语言处理、搜索索引等。在本指南中，我们将探讨 LangChain 提供的三种不同的文本分割器，可用于有效地分割 HTML 内容：

HTMLHeaderTextSplitter
HTMLSectionSplitter
HTMLSemanticPreservingSplitter

这些分割器 each 都有独特的功能和用例。本指南将帮助您了解它们之间的区别、为何选择其中一个而非其他，以及如何有效地使用它们。

%pip install -qU langchain-text-splitters

分割器概览

HTMLHeaderTextSplitter

info

当您想根据文档的标题保留其分层结构时，此功能非常有用。

描述: 根据标题标签（例如 <h1>、<h2>、<h3> 等）分割 HTML 文本，并为每个标题添加与给定块相关的元数据。

功能:

在 HTML 元素级别分割文本。
保留编码在文档结构中的富含上下文的信息。
可以逐个元素返回块，或将具有相同元数据的元素组合在一起。

HTMLSectionSplitter

info

当您希望将 HTML 文档拆分为更大的部分时很有用，例如 <section>, <div> 或自定义定义的节。

描述: 与 HTMLHeaderTextSplitter 类似，但侧重于根据指定的标签将 HTML 拆分为各个部分。

功能:

使用 XSLT 转换来检测和拆分部分。
内部使用 RecursiveCharacterTextSplitter 处理大型部分。
考虑字体大小来确定部分。

HTMLSemanticPreservingSplitter

info

当您需要确保结构化元素不会在块之间被分割，从而保持上下文相关性时，这是理想选择。

Description: 在将 HTML 内容分割成可管理块的同时，保留表格、列表和其他 HTML 组件等重要元素的语义结构。

Capabilities:

保留表格、列表和其他指定的 HTML 元素。
允许为特定的 HTML 标签使用自定义处理程序。
确保文档的语义含义得以维持。
内置标准化和停用词移除功能

选择合适的分隔器

当您需要根据 HTML 文档的标题层级进行分割，并保留关于标题的元数据时，请使用 HTMLHeaderTextSplitter 。
当您需要将文档分割成更大、更通用的部分，可能基于自定义标签或字体大小时，请使用 HTMLSectionSplitter 。
当您需要将文档分割成块，同时保留表格和列表等语义化元素，确保它们不会被分割并且其上下文得以维持时，请使用 HTMLSemanticPreservingSplitter 。

功能	HTMLHeaderTextSplitter	HTMLSectionSplitter	HTMLSemanticPreservingSplitter
基于标题进行分割	Yes	Yes	Yes
保留语义化元素 (表格、列表)	No	No	Yes
为标题添加元数据	Yes	Yes	Yes
为 HTML 标签提供自定义处理器	No	No	Yes
保留媒体 (图片、视频)	No	No	Yes
考虑字体大小	No	Yes	No
使用 XSLT 转换	No	Yes	No

示例 HTML 文档

让我们使用以下 HTML 文档作为示例：

html_string = """
<!DOCTYPE html>
  <html lang='en'>
  <head>
    <meta charset='UTF-8'>
    <meta name='viewport' content='width=device-width, initial-scale=1.0'>
    <title>Fancy Example HTML Page</title>
  </head>
  <body>
    <h1>Main Title</h1>
    <p>This is an introductory paragraph with some basic content.</p>
    
    <h2>Section 1: Introduction</h2>
    <p>This section introduces the topic. Below is a list:</p>
    <ul>
      <li>First item</li>
      <li>Second item</li>
      <li>Third item with <strong>bold text</strong> and <a href='#'>a link</a></li>
    </ul>
    
    <h3>Subsection 1.1: Details</h3>
    <p>This subsection provides additional details. Here's a table:</p>
    <table border='1'>
      <thead>
        <tr>
          <th>Header 1</th>
          <th>Header 2</th>
          <th>Header 3</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <td>Row 1, Cell 1</td>
          <td>Row 1, Cell 2</td>
          <td>Row 1, Cell 3</td>
        </tr>
        <tr>
          <td>Row 2, Cell 1</td>
          <td>Row 2, Cell 2</td>
          <td>Row 2, Cell 3</td>
        </tr>
      </tbody>
    </table>
    
    <h2>Section 2: Media Content</h2>
    <p>This section contains an image and a video:</p>
      <img src='example_image_link.mp4' alt='Example Image'>
      <video controls width='250' src='example_video_link.mp4' type='video/mp4'>
      Your browser does not support the video tag.
    </video>

    <h2>Section 3: Code Example</h2>
    <p>This section contains a code block:</p>
    <pre><code data-lang="html">
    &lt;div&gt;
      &lt;p&gt;This is a paragraph inside a div.&lt;/p&gt;
    &lt;/div&gt;
    </code></pre>

    <h2>Conclusion</h2>
    <p>This is the conclusion of the document.</p>
  </body>
  </html>
"""

使用 HTMLHeaderTextSplitter

HTMLHeaderTextSplitter 是一个“结构感知”的文本分割器，它在 HTML 元素级别分割文本，并为与给定块“相关”的每个标题添加元数据。它可以逐个元素返回块，或将具有相同元数据的元素合并在一起，其目标是（a）（或多或少）在语义上将相关文本分组，以及（b）保留文档结构中编码的富含上下文的信息。它可以作为分块管道的一部分与其他文本分割器一起使用。

它类似于 Markdown 文件的 MarkdownHeaderTextSplitter。

要指定在哪些标题上进行分割，请在实例化 HTMLHeaderTextSplitter 时指定 headers_to_split_on，如下所示。

from langchain_text_splitters import HTMLHeaderTextSplitter

headers_to_split_on = [
    ("h1", "Header 1"),
    ("h2", "Header 2"),
    ("h3", "Header 3"),
]

html_splitter = HTMLHeaderTextSplitter(headers_to_split_on)
html_header_splits = html_splitter.split_text(html_string)
html_header_splits

API Reference:HTMLHeaderTextSplitter

[Document(metadata={'Header 1': 'Main Title'}, page_content='This is an introductory paragraph with some basic content.'),
 Document(metadata={'Header 1': 'Main Title', 'Header 2': 'Section 1: Introduction'}, page_content='This section introduces the topic. Below is a list:  \nFirst item Second item Third item with bold text and a link'),
 Document(metadata={'Header 1': 'Main Title', 'Header 2': 'Section 1: Introduction', 'Header 3': 'Subsection 1.1: Details'}, page_content="This subsection provides additional details. Here's a table:"),
 Document(metadata={'Header 1': 'Main Title', 'Header 2': 'Section 2: Media Content'}, page_content='This section contains an image and a video:'),
 Document(metadata={'Header 1': 'Main Title', 'Header 2': 'Section 3: Code Example'}, page_content='This section contains a code block:'),
 Document(metadata={'Header 1': 'Main Title', 'Header 2': 'Conclusion'}, page_content='This is the conclusion of the document.')]

HTMLHeaderTextSplitter 的实例化中指定 return_each_element=True，以将每个元素及其关联的标题一并返回：

html_splitter = HTMLHeaderTextSplitter(
    headers_to_split_on,
    return_each_element=True,
)
html_header_splits_elements = html_splitter.split_text(html_string)

与上面的内容进行比较，元素是按其标题聚合的：

for element in html_header_splits[:2]:
    print(element)

page_content='This is an introductory paragraph with some basic content.' metadata={'Header 1': 'Main Title'}
page_content='This section introduces the topic. Below is a list:  
First item Second item Third item with bold text and a link' metadata={'Header 1': 'Main Title', 'Header 2': 'Section 1: Introduction'}

现在每个元素都作为独立的 Document 返回：

for element in html_header_splits_elements[:3]:
    print(element)

page_content='This is an introductory paragraph with some basic content.' metadata={'Header 1': 'Main Title'}
page_content='This section introduces the topic. Below is a list:' metadata={'Header 1': 'Main Title', 'Header 2': 'Section 1: Introduction'}
page_content='First item Second item Third item with bold text and a link' metadata={'Header 1': 'Main Title', 'Header 2': 'Section 1: Introduction'}

如何从 URL 或 HTML 文件中拆分文本：

要直接从 URL 读取，将 URL 字符串传递给 split_text_from_url 方法。

同样，本地 HTML 文件可以传递给 split_text_from_file 方法。

url = "https://plato.stanford.edu/entries/goedel/"

headers_to_split_on = [
    ("h1", "Header 1"),
    ("h2", "Header 2"),
    ("h3", "Header 3"),
    ("h4", "Header 4"),
]

html_splitter = HTMLHeaderTextSplitter(headers_to_split_on)

# for local file use html_splitter.split_text_from_file(<path_to_file>)
html_header_splits = html_splitter.split_text_from_url(url)

如何约束分块大小：

HTMLHeaderTextSplitter（基于 HTML 头部进行分割）可以与另一个约束分割基于字符长度的分隔器组合使用，例如 RecursiveCharacterTextSplitter。

这可以通过第二个分割器的 .split_documents 方法来实现：

from langchain_text_splitters import RecursiveCharacterTextSplitter

chunk_size = 500
chunk_overlap = 30
text_splitter = RecursiveCharacterTextSplitter(
    chunk_size=chunk_size, chunk_overlap=chunk_overlap
)

# Split
splits = text_splitter.split_documents(html_header_splits)
splits[80:85]

API Reference:RecursiveCharacterTextSplitter

[Document(metadata={'Header 1': 'Kurt Gödel', 'Header 2': '2. Gödel’s Mathematical Work', 'Header 3': '2.2 The Incompleteness Theorems', 'Header 4': '2.2.1 The First Incompleteness Theorem'}, page_content='We see that Gödel first tried to reduce the consistency problem for analysis to that of arithmetic. This seemed to require a truth definition for arithmetic, which in turn led to paradoxes, such as the Liar paradox (“This sentence is false”) and Berry’s paradox (“The least number not defined by an expression consisting of just fourteen English words”). Gödel then noticed that such paradoxes would not necessarily arise if truth were replaced by provability. But this means that arithmetic truth'),
 Document(metadata={'Header 1': 'Kurt Gödel', 'Header 2': '2. Gödel’s Mathematical Work', 'Header 3': '2.2 The Incompleteness Theorems', 'Header 4': '2.2.1 The First Incompleteness Theorem'}, page_content='means that arithmetic truth and arithmetic provability are not co-extensive — whence the First Incompleteness Theorem.'),
 Document(metadata={'Header 1': 'Kurt Gödel', 'Header 2': '2. Gödel’s Mathematical Work', 'Header 3': '2.2 The Incompleteness Theorems', 'Header 4': '2.2.1 The First Incompleteness Theorem'}, page_content='This account of Gödel’s discovery was told to Hao Wang very much after the fact; but in Gödel’s contemporary correspondence with Bernays and Zermelo, essentially the same description of his path to the theorems is given. (See Gödel 2003a and Gödel 2003b respectively.) From those accounts we see that the undefinability of truth in arithmetic, a result credited to Tarski, was likely obtained in some form by Gödel by 1931. But he neither publicized nor published the result; the biases logicians'),
 Document(metadata={'Header 1': 'Kurt Gödel', 'Header 2': '2. Gödel’s Mathematical Work', 'Header 3': '2.2 The Incompleteness Theorems', 'Header 4': '2.2.1 The First Incompleteness Theorem'}, page_content='result; the biases logicians had expressed at the time concerning the notion of truth, biases which came vehemently to the fore when Tarski announced his results on the undefinability of truth in formal systems 1935, may have served as a deterrent to Gödel’s publication of that theorem.'),
 Document(metadata={'Header 1': 'Kurt Gödel', 'Header 2': '2. Gödel’s Mathematical Work', 'Header 3': '2.2 The Incompleteness Theorems', 'Header 4': '2.2.2 The proof of the First Incompleteness Theorem'}, page_content='We now describe the proof of the two theorems, formulating Gödel’s results in Peano arithmetic. Gödel himself used a system related to that defined in Principia Mathematica, but containing Peano arithmetic. In our presentation of the First and Second Incompleteness Theorems we refer to Peano arithmetic as P, following Gödel’s notation.')]

局限性

HTML 文档的结构可能存在很大的变异性，虽然 HTMLHeaderTextSplitter 会尝试将所有“相关”的标题附加到给定的块中，但有时它可能会遗漏某些标题。例如，该算法假定一种信息层级结构，其中标题始终位于与其关联文本“之上”的节点上，即前面的同级节点、祖节点或它们的组合。在以下新闻文章（截至本文撰写时）中，文档的结构使得顶层标题的文本虽然带有“h1”标签，但它与我们期望它应该“之上”的文本元素位于一个不同的子树中—因此，我们可以观察到“h1”元素及其关联文本没有出现在块元数据中（但，在适用的情况下，我们会看到“h2”及其关联文本）：

url = "https://www.cnn.com/2023/09/25/weather/el-nino-winter-us-climate/index.html"

headers_to_split_on = [
    ("h1", "Header 1"),
    ("h2", "Header 2"),
]

html_splitter = HTMLHeaderTextSplitter(headers_to_split_on)
html_header_splits = html_splitter.split_text_from_url(url)
print(html_header_splits[1].page_content[:500])

No two El Niño winters are the same, but many have temperature and precipitation trends in common.  
Average conditions during an El Niño winter across the continental US.  
One of the major reasons is the position of the jet stream, which often shifts south during an El Niño winter. This shift typically brings wetter and cooler weather to the South while the North becomes drier and warmer, according to NOAA.  
Because the jet stream is essentially a river of air that storms flow through, they c

使用 HTMLSectionSplitter

与 HTMLHeaderTextSplitter 的概念类似，HTMLSectionSplitter 是一个“感知结构”的文本分割器，它在元素级别分割文本，并为“相关”于任何给定块的标题添加元数据。它允许您按节分割 HTML。

它可以逐个元素地返回块，或者组合具有相同元数据的元素，目标是 (a) 将相关的文本（或多或少）在语义上分组，以及 (b) 保留文档结构中编码的富含上下文的信息。

使用 xslt_path 提供一个绝对路径来转换 HTML，以便它可以根据提供的标签检测节。默认是使用 data_connection/document_transformers 目录中的 converting_to_header.xslt 文件。这是为了将 HTML 转换为更容易检测节的格式/布局。例如，基于字体大小的 span 可以转换为标题标签以被检测为节。

如何分割 HTML 字符串：

from langchain_text_splitters import HTMLSectionSplitter

headers_to_split_on = [
    ("h1", "Header 1"),
    ("h2", "Header 2"),
]

html_splitter = HTMLSectionSplitter(headers_to_split_on)
html_header_splits = html_splitter.split_text(html_string)
html_header_splits

API Reference:HTMLSectionSplitter

[Document(metadata={'Header 1': 'Main Title'}, page_content='Main Title \n This is an introductory paragraph with some basic content.'),
 Document(metadata={'Header 2': 'Section 1: Introduction'}, page_content="Section 1: Introduction \n This section introduces the topic. Below is a list: \n \n First item \n Second item \n Third item with  bold text  and  a link \n \n \n Subsection 1.1: Details \n This subsection provides additional details. Here's a table: \n \n \n \n Header 1 \n Header 2 \n Header 3 \n \n \n \n \n Row 1, Cell 1 \n Row 1, Cell 2 \n Row 1, Cell 3 \n \n \n Row 2, Cell 1 \n Row 2, Cell 2 \n Row 2, Cell 3"),
 Document(metadata={'Header 2': 'Section 2: Media Content'}, page_content='Section 2: Media Content \n This section contains an image and a video: \n \n \n      Your browser does not support the video tag.'),
 Document(metadata={'Header 2': 'Section 3: Code Example'}, page_content='Section 3: Code Example \n This section contains a code block: \n \n    <div>\n      <p>This is a paragraph inside a div.</p>\n    </div>'),
 Document(metadata={'Header 2': 'Conclusion'}, page_content='Conclusion \n This is the conclusion of the document.')]

如何限制块大小：

HTMLSectionSplitter 可与其他文本分割器结合使用，作为分块管道的一部分。在内部，当区块大小大于块大小时，它会使用 RecursiveCharacterTextSplitter。它还会考虑文本的字体大小，以根据确定的字体大小阈值确定它是否为区块。

from langchain_text_splitters import RecursiveCharacterTextSplitter

headers_to_split_on = [
    ("h1", "Header 1"),
    ("h2", "Header 2"),
    ("h3", "Header 3"),
]

html_splitter = HTMLSectionSplitter(headers_to_split_on)

html_header_splits = html_splitter.split_text(html_string)

chunk_size = 50
chunk_overlap = 5
text_splitter = RecursiveCharacterTextSplitter(
    chunk_size=chunk_size, chunk_overlap=chunk_overlap
)

# Split
splits = text_splitter.split_documents(html_header_splits)
splits

API Reference:RecursiveCharacterTextSplitter

[Document(metadata={'Header 1': 'Main Title'}, page_content='Main Title'),
 Document(metadata={'Header 1': 'Main Title'}, page_content='This is an introductory paragraph with some'),
 Document(metadata={'Header 1': 'Main Title'}, page_content='some basic content.'),
 Document(metadata={'Header 2': 'Section 1: Introduction'}, page_content='Section 1: Introduction'),
 Document(metadata={'Header 2': 'Section 1: Introduction'}, page_content='This section introduces the topic. Below is a'),
 Document(metadata={'Header 2': 'Section 1: Introduction'}, page_content='is a list:'),
 Document(metadata={'Header 2': 'Section 1: Introduction'}, page_content='First item \n Second item'),
 Document(metadata={'Header 2': 'Section 1: Introduction'}, page_content='Third item with  bold text  and  a link'),
 Document(metadata={'Header 3': 'Subsection 1.1: Details'}, page_content='Subsection 1.1: Details'),
 Document(metadata={'Header 3': 'Subsection 1.1: Details'}, page_content='This subsection provides additional details.'),
 Document(metadata={'Header 3': 'Subsection 1.1: Details'}, page_content="Here's a table:"),
 Document(metadata={'Header 3': 'Subsection 1.1: Details'}, page_content='Header 1 \n Header 2 \n Header 3'),
 Document(metadata={'Header 3': 'Subsection 1.1: Details'}, page_content='Row 1, Cell 1 \n Row 1, Cell 2'),
 Document(metadata={'Header 3': 'Subsection 1.1: Details'}, page_content='Row 1, Cell 3 \n \n \n Row 2, Cell 1'),
 Document(metadata={'Header 3': 'Subsection 1.1: Details'}, page_content='Row 2, Cell 2 \n Row 2, Cell 3'),
 Document(metadata={'Header 2': 'Section 2: Media Content'}, page_content='Section 2: Media Content'),
 Document(metadata={'Header 2': 'Section 2: Media Content'}, page_content='This section contains an image and a video:'),
 Document(metadata={'Header 2': 'Section 2: Media Content'}, page_content='Your browser does not support the video'),
 Document(metadata={'Header 2': 'Section 2: Media Content'}, page_content='tag.'),
 Document(metadata={'Header 2': 'Section 3: Code Example'}, page_content='Section 3: Code Example'),
 Document(metadata={'Header 2': 'Section 3: Code Example'}, page_content='This section contains a code block: \n \n    <div>'),
 Document(metadata={'Header 2': 'Section 3: Code Example'}, page_content='<p>This is a paragraph inside a div.</p>'),
 Document(metadata={'Header 2': 'Section 3: Code Example'}, page_content='</div>'),
 Document(metadata={'Header 2': 'Conclusion'}, page_content='Conclusion'),
 Document(metadata={'Header 2': 'Conclusion'}, page_content='This is the conclusion of the document.')]

使用 HTMLSemanticPreservingSplitter

HTMLSemanticPreservingSplitter 旨在将 HTML 内容分割成易于管理的代码块，同时保留表格、列表和其他 HTML 组件等重要元素的语义结构。这确保了这些元素不会跨越代码块被分割，从而避免了表格头、列表头等上下文相关性的丢失。

此分割器的核心设计目标是创建具有上下文相关性的代码块。使用 HTMLHeaderTextSplitter 进行的通用递归分割可能会将表格、列表和其他结构化元素分割到中间，丢失大量上下文并创建不好的代码块。

HTMLSemanticPreservingSplitter 对于分割包含表格和列表等结构化元素的 HTML 内容至关重要，尤其是在保持这些元素完整性至关重要的情况下。此外，它能够为特定 HTML 标签定义自定义处理程序，使其成为处理复杂 HTML 文档的通用工具。

重要提示：max_chunk_size 不是代码块的绝对最大尺寸，最大尺寸的计算发生在保留的内容不是代码块的一部分时，以确保其不被分割。当我们添加保留的数据回代码块时，代码块的尺寸有可能超过 max_chunk_size。这对于确保我们保持原始文档的结构至关重要。

info

说明：

我们定义了一个自定义处理程序来重新格式化代码块的内容
我们为特定的 html 元素定义了一个禁止列表，以便在预处理之前对其及其内容进行分解
我们有意设置了一个较小的代码块大小，以演示元素的非分割性

# BeautifulSoup is required to use the custom handlers
from bs4 import Tag
from langchain_text_splitters import HTMLSemanticPreservingSplitter

headers_to_split_on = [
    ("h1", "Header 1"),
    ("h2", "Header 2"),
]


def code_handler(element: Tag) -> str:
    data_lang = element.get("data-lang")
    code_format = f"<code:{data_lang}>{element.get_text()}</code>"

    return code_format


splitter = HTMLSemanticPreservingSplitter(
    headers_to_split_on=headers_to_split_on,
    separators=["\n\n", "\n", ". ", "! ", "? "],
    max_chunk_size=50,
    preserve_images=True,
    preserve_videos=True,
    elements_to_preserve=["table", "ul", "ol", "code"],
    denylist_tags=["script", "style", "head"],
    custom_handlers={"code": code_handler},
)

documents = splitter.split_text(html_string)
documents

API Reference:HTMLSemanticPreservingSplitter

[Document(metadata={'Header 1': 'Main Title'}, page_content='This is an introductory paragraph with some basic content.'),
 Document(metadata={'Header 2': 'Section 1: Introduction'}, page_content='This section introduces the topic'),
 Document(metadata={'Header 2': 'Section 1: Introduction'}, page_content='. Below is a list: First item Second item Third item with bold text and a link Subsection 1.1: Details This subsection provides additional details'),
 Document(metadata={'Header 2': 'Section 1: Introduction'}, page_content=". Here's a table: Header 1 Header 2 Header 3 Row 1, Cell 1 Row 1, Cell 2 Row 1, Cell 3 Row 2, Cell 1 Row 2, Cell 2 Row 2, Cell 3"),
 Document(metadata={'Header 2': 'Section 2: Media Content'}, page_content='This section contains an image and a video: ![image:example_image_link.mp4](example_image_link.mp4) ![video:example_video_link.mp4](example_video_link.mp4)'),
 Document(metadata={'Header 2': 'Section 3: Code Example'}, page_content='This section contains a code block: <code:html> <div> <p>This is a paragraph inside a div.</p> </div> </code>'),
 Document(metadata={'Header 2': 'Conclusion'}, page_content='This is the conclusion of the document.')]

保留表格和列表

在此示例中，我们将演示 HTMLSemanticPreservingSplitter 如何保留 HTML 文档中的表格和大型列表。块大小将设置为 50 个字符，以说明即使这些元素超过了定义的块大小限制，分割器也能确保它们不被分割。

from langchain_text_splitters import HTMLSemanticPreservingSplitter

html_string = """
<!DOCTYPE html>
<html>
    <body>
        <div>
            <h1>Section 1</h1>
            <p>This section contains an important table and list that should not be split across chunks.</p>
            <table>
                <tr>
                    <th>Item</th>
                    <th>Quantity</th>
                    <th>Price</th>
                </tr>
                <tr>
                    <td>Apples</td>
                    <td>10</td>
                    <td>$1.00</td>
                </tr>
                <tr>
                    <td>Oranges</td>
                    <td>5</td>
                    <td>$0.50</td>
                </tr>
                <tr>
                    <td>Bananas</td>
                    <td>50</td>
                    <td>$1.50</td>
                </tr>
            </table>
            <h2>Subsection 1.1</h2>
            <p>Additional text in subsection 1.1 that is separated from the table and list.</p>
            <p>Here is a detailed list:</p>
            <ul>
                <li>Item 1: Description of item 1, which is quite detailed and important.</li>
                <li>Item 2: Description of item 2, which also contains significant information.</li>
                <li>Item 3: Description of item 3, another item that we don't want to split across chunks.</li>
            </ul>
        </div>
    </body>
</html>
"""

headers_to_split_on = [("h1", "Header 1"), ("h2", "Header 2")]

splitter = HTMLSemanticPreservingSplitter(
    headers_to_split_on=headers_to_split_on,
    max_chunk_size=50,
    elements_to_preserve=["table", "ul"],
)

documents = splitter.split_text(html_string)
print(documents)

API Reference:HTMLSemanticPreservingSplitter

[Document(metadata={'Header 1': 'Section 1'}, page_content='This section contains an important table and list'), Document(metadata={'Header 1': 'Section 1'}, page_content='that should not be split across chunks.'), Document(metadata={'Header 1': 'Section 1'}, page_content='Item Quantity Price Apples 10 $1.00 Oranges 5 $0.50 Bananas 50 $1.50'), Document(metadata={'Header 2': 'Subsection 1.1'}, page_content='Additional text in subsection 1.1 that is'), Document(metadata={'Header 2': 'Subsection 1.1'}, page_content='separated from the table and list. Here is a'), Document(metadata={'Header 2': 'Subsection 1.1'}, page_content="detailed list: Item 1: Description of item 1, which is quite detailed and important. Item 2: Description of item 2, which also contains significant information. Item 3: Description of item 3, another item that we don't want to split across chunks.")]

说明

在此示例中，HTMLSemanticPreservingSplitter 确保整个表格和无序列表 (<ul>) 都保留在其各自的代码块中。即使代码块大小设置为 50 个字符，该分割器也能识别这些元素不应该被拆分，并保持它们的完整性。

这在处理数据表格或列表时尤为重要，因为拆分内容可能导致上下文丢失或混淆。生成的 Document 对象保留了这些元素的完整结构，确保信息的上下文相关性得以维持。

使用自定义处理器

HTMLSemanticPreservingSplitter 允许你为特定的 HTML 元素定义自定义处理器。某些平台拥有 BeautifulSoup 无法原生解析的自定义 HTML 标签，此时，你可以利用自定义处理器轻松添加格式化逻辑。

这对于需要特殊处理的元素特别有用，例如 <iframe> 标签或特定的 data- 元素。在本例中，我们将为 iframe 标签创建一个自定义处理器，将其转换为类似 Markdown 的链接。

def custom_iframe_extractor(iframe_tag):
    iframe_src = iframe_tag.get("src", "")
    return f"[iframe:{iframe_src}]({iframe_src})"


splitter = HTMLSemanticPreservingSplitter(
    headers_to_split_on=headers_to_split_on,
    max_chunk_size=50,
    separators=["\n\n", "\n", ". "],
    elements_to_preserve=["table", "ul", "ol"],
    custom_handlers={"iframe": custom_iframe_extractor},
)

html_string = """
<!DOCTYPE html>
<html>
    <body>
        <div>
            <h1>Section with Iframe</h1>
            <iframe src="https://example.com/embed"></iframe>
            <p>Some text after the iframe.</p>
            <ul>
                <li>Item 1: Description of item 1, which is quite detailed and important.</li>
                <li>Item 2: Description of item 2, which also contains significant information.</li>
                <li>Item 3: Description of item 3, another item that we don't want to split across chunks.</li>
            </ul>
        </div>
    </body>
</html>
"""

documents = splitter.split_text(html_string)
print(documents)

[Document(metadata={'Header 1': 'Section with Iframe'}, page_content='[iframe:https://example.com/embed](https://example.com/embed) Some text after the iframe'), Document(metadata={'Header 1': 'Section with Iframe'}, page_content=". Item 1: Description of item 1, which is quite detailed and important. Item 2: Description of item 2, which also contains significant information. Item 3: Description of item 3, another item that we don't want to split across chunks.")]

说明

在此示例中，我们为 iframe 标签定义了一个自定义处理器，将其转换为类似 Markdown 的链接。当 splitter 处理 HTML 内容时，它会使用此自定义处理器来转换 iframe 标签，同时保留表格和列表等其他元素。生成的 Document 对象展示了根据您提供的自定义逻辑如何处理 iframe。

重要提示: 在保留链接等项目时，请注意不要在分隔符中包含 .，或将分隔符留空。RecursiveCharacterTextSplitter 会在句号处拆分，这会把链接一分为二。请确保提供一个分隔符列表，其中包含 . 以代替。

使用自定义处理程序通过 LLM 分析图像

借助自定义处理程序，我们还可以覆盖任何元素的默认处理方式。一个很好的例子是在文档中插入图像的语义分析，直接在分块流程中进行。

由于我们的函数在发现标签时被调用，我们可以覆盖 <img> 标签并关闭 preserve_images，以便插入我们希望嵌入到分块中的任何内容。

"""此示例假设您有辅助方法 `load_image_from_url` 和一个可以处理图像数据的 LLM 代理 `llm`。"""

from langchain.agents import AgentExecutor

# 此示例需要替换为您自己的代理
llm = AgentExecutor(...)


# 此方法是用于从 URL 加载图像数据的占位符，此处未实现
def load_image_from_url(image_url: str) -> bytes:
    # 假设此方法从 URL 获取图像数据
    return b"image_data"


html_string = """
<!DOCTYPE html>
<html>
    <body>
        <div>
            <h1>Section with Image and Link</h1>
            <p>
                <img src="https://example.com/image.jpg" alt="An example image" />
                Some text after the image.
            </p>
            <ul>
                <li>Item 1: Description of item 1, which is quite detailed and important.</li>
                <li>Item 2: Description of item 2, which also contains significant information.</li>
                <li>Item 3: Description of item 3, another item that we don't want to split across chunks.</li>
            </ul>
        </div>
    </body>
</html>
"""


def custom_image_handler(img_tag) -> str:
    img_src = img_tag.get("src", "")
    img_alt = img_tag.get("alt", "No alt text provided")

    image_data = load_image_from_url(img_src)
    semantic_meaning = llm.invoke(image_data)

    markdown_text = f"[Image Alt Text: {img_alt} | Image Source: {img_src} | Image Semantic Meaning: {semantic_meaning}]"

    return markdown_text


splitter = HTMLSemanticPreservingSplitter(
    headers_to_split_on=headers_to_split_on,
    max_chunk_size=50,
    separators=["\n\n", "\n", ". "],
    elements_to_preserve=["ul"],
    preserve_images=False,
    custom_handlers={"img": custom_image_handler},
)

documents = splitter.split_text(html_string)

print(documents)

API Reference:AgentExecutor

[Document(metadata={'Header 1': 'Section with Image and Link'}, page_content='[Image Alt Text: An example image | Image Source: https://example.com/image.jpg | Image Semantic Meaning: semantic-meaning] Some text after the image'), 
Document(metadata={'Header 1': 'Section with Image and Link'}, page_content=". Item 1: Description of item 1, which is quite detailed and important. Item 2: Description of item 2, which also contains significant information. Item 3: Description of item 3, another item that we don't want to split across chunks.")]

解释：

通过我们编写的自定义处理程序，用于从 HTML 中的 <img> 元素提取特定字段，我们可以使用我们的代理进一步处理数据，并将结果直接插入到我们的块中。请务必确保 preserve_images 设置为 False，否则将执行默认的 <img> 字段处理。

分割器概览​

HTMLHeaderTextSplitter​

HTMLSectionSplitter​

HTMLSemanticPreservingSplitter​

选择合适的分隔器​

示例 HTML 文档​

使用 HTMLHeaderTextSplitter​

如何从 URL 或 HTML 文件中拆分文本：​

如何约束分块大小：​

局限性​

使用 HTMLSectionSplitter​

如何分割 HTML 字符串：​

如何限制块大小：​

使用 HTMLSemanticPreservingSplitter​

保留表格和列表​

说明​

使用自定义处理器​

说明​

使用自定义处理程序通过 LLM 分析图像​

解释：​

分割器概览

HTMLHeaderTextSplitter

HTMLSectionSplitter

HTMLSemanticPreservingSplitter

选择合适的分隔器

示例 HTML 文档

使用 HTMLHeaderTextSplitter

如何从 URL 或 HTML 文件中拆分文本：

如何约束分块大小：

局限性

使用 HTMLSectionSplitter

如何分割 HTML 字符串：

如何限制块大小：

使用 HTMLSemanticPreservingSplitter

保留表格和列表

说明

使用自定义处理器

说明

使用自定义处理程序通过 LLM 分析图像

解释：