Timescale Vector (Postgres)

Timescale Vector 是专为 AI 应用打造的 PostgreSQL++。它使您能够在 PostgreSQL 中高效地存储和查询数十亿个向量嵌入。

PostgreSQL 也称为 Postgres，是一个免费开源的关系数据库管理系统 (RDBMS)，它强调可扩展性和 SQL 的兼容性。

本笔记本展示了如何使用 Postgres 向量数据库 (TimescaleVector) 进行自查询。在笔记本中，我们将演示围绕 TimescaleVector 向量存储包装的 SelfQueryRetriever。

Timescale Vector 是什么？

Timescale Vector 是专为 AI 应用打造的 PostgreSQL++。

Timescale Vector 使您能够在 PostgreSQL 中高效地存储和查询数百万个向量嵌入。

通过受 DiskANN 启发的索引算法，在超过 10 亿个向量上实现更快、更准确的相似性搜索，从而增强 pgvector。
通过自动基于时间的分割和索引实现快速的时间序列向量搜索。
提供熟悉的 SQL 接口，用于查询向量嵌入和关系数据。

Timescale Vector 是云端 PostgreSQL，专为 AI 应用而设计，可与您共同扩展，从 POC 到生产环境：

通过使您能够在单个数据库中存储关系元数据、向量嵌入和时间序列数据来简化操作。
受益于稳固的 PostgreSQL 基础，具备企业级功能，如流式备份和复制、高可用性和行级安全性。
通过企业级安全性和合规性，带来无忧体验。

如何访问 Timescale Vector

Timescale Vector 可在 Timescale 平台上获得，该平台是云端 PostgreSQL 平台。（目前没有自托管版本。）

LangChain 用户可免费试用 Timescale Vector 90 天。

要开始使用，请在 Timescale 注册，创建一个新数据库，然后按照本笔记本操作！
有关更多详细信息和性能基准测试，请参阅 Timescale Vector 解释博客。
有关在 Python 中使用 Timescale Vector 的更多详细信息，请参阅安装说明。

创建 TimescaleVector vectorstore

首先，我们将创建一个 Timescale Vector vectorstore 并向其中填充数据。我们创建了一个包含电影摘要的小型演示文档集。

注意：自查询检索器要求您安装 lark（pip install lark）。我们还需要 timescale-vector 包。

%pip install --upgrade --quiet  lark

%pip install --upgrade --quiet  timescale-vector

在此示例中，我们将使用 OpenAIEmbeddings，因此让我们加载您的 OpenAI API 密钥。

# Get openAI api key by reading local .env file
# The .env file should contain a line starting with `OPENAI_API_KEY=sk-`
import os

from dotenv import find_dotenv, load_dotenv

_ = load_dotenv(find_dotenv())

OPENAI_API_KEY = os.environ["OPENAI_API_KEY"]
# Alternatively, use getpass to enter the key in a prompt
# import os
# import getpass
# os.environ["OPENAI_API_KEY"] = getpass.getpass("OpenAI API Key:")

要连接到您的 PostgreSQL 数据库，您需要服务 URI，可以在创建新数据库后下载的备忘单或 .env 文件中找到。

如果您还没有，请注册 Timescale 并创建一个新数据库。

URI 看起来会像这样：postgres://tsdbadmin:<password>@<id>.tsdb.cloud.timescale.com:<port>/tsdb?sslmode=require

# Get the service url by reading local .env file
# The .env file should contain a line starting with `TIMESCALE_SERVICE_URL=postgresql://`
_ = load_dotenv(find_dotenv())
TIMESCALE_SERVICE_URL = os.environ["TIMESCALE_SERVICE_URL"]

# Alternatively, use getpass to enter the key in a prompt
# import os
# import getpass
# TIMESCALE_SERVICE_URL = getpass.getpass("Timescale Service URL:")

from langchain_community.vectorstores.timescalevector import TimescaleVector
from langchain_core.documents import Document
from langchain_openai import OpenAIEmbeddings

embeddings = OpenAIEmbeddings()

API Reference:TimescaleVector | Document | OpenAIEmbeddings

这里是我们将在本次演示中使用的一些示例文档。数据内容是关于电影的，包含内容和元数据字段，其中存储着特定电影的信息。

docs = [
    Document(
        page_content="A bunch of scientists bring back dinosaurs and mayhem breaks loose",
        metadata={"year": 1993, "rating": 7.7, "genre": "science fiction"},
    ),
    Document(
        page_content="Leo DiCaprio gets lost in a dream within a dream within a dream within a ...",
        metadata={"year": 2010, "director": "Christopher Nolan", "rating": 8.2},
    ),
    Document(
        page_content="A psychologist / detective gets lost in a series of dreams within dreams within dreams and Inception reused the idea",
        metadata={"year": 2006, "director": "Satoshi Kon", "rating": 8.6},
    ),
    Document(
        page_content="A bunch of normal-sized women are supremely wholesome and some men pine after them",
        metadata={"year": 2019, "director": "Greta Gerwig", "rating": 8.3},
    ),
    Document(
        page_content="Toys come alive and have a blast doing so",
        metadata={"year": 1995, "genre": "animated"},
    ),
    Document(
        page_content="Three men walk into the Zone, three men walk out of the Zone",
        metadata={
            "year": 1979,
            "director": "Andrei Tarkovsky",
            "genre": "science fiction",
            "rating": 9.9,
        },
    ),
]

最后，我们将创建 Timescale Vector vectorstore。请注意，collection name 将是文档存储在 PostgreSQL 表中的表名。

COLLECTION_NAME = "langchain_self_query_demo"
vectorstore = TimescaleVector.from_documents(
    embedding=embeddings,
    documents=docs,
    collection_name=COLLECTION_NAME,
    service_url=TIMESCALE_SERVICE_URL,
)

创建自查询检索器

现在我们可以实例化我们的检索器了。要做到这一点，我们需要提前提供一些关于我们的文档所支持的元数据字段以及文档内容的简短描述的信息。

from langchain.chains.query_constructor.schema import AttributeInfo
from langchain.retrievers.self_query.base import SelfQueryRetriever
from langchain_openai import OpenAI

# Give LLM info about the metadata fields
metadata_field_info = [
    AttributeInfo(
        name="genre",
        description="The genre of the movie",
        type="string or list[string]",
    ),
    AttributeInfo(
        name="year",
        description="The year the movie was released",
        type="integer",
    ),
    AttributeInfo(
        name="director",
        description="The name of the movie director",
        type="string",
    ),
    AttributeInfo(
        name="rating", description="A 1-10 rating for the movie", type="float"
    ),
]
document_content_description = "Brief summary of a movie"

# Instantiate the self-query retriever from an LLM
llm = OpenAI(temperature=0)
retriever = SelfQueryRetriever.from_llm(
    llm, vectorstore, document_content_description, metadata_field_info, verbose=True
)

API Reference:AttributeInfo | SelfQueryRetriever | OpenAI

使用 Timescale Vector 进行的自查询检索

现在我们可以实际使用我们的检索器了！

运行下面的查询，并注意你可以用自然语言指定查询、过滤器、复合过滤器（带 AND、OR 的过滤器），自查询检索器会将该查询翻译成 SQL，并在 Timescale Vector (Postgres) 向量数据库上执行搜索。

这说明了自查询检索器的强大之处。你可以使用它对你的向量数据库执行复杂的搜索，而你或你的用户无需直接编写任何 SQL！

# This example only specifies a relevant query
retriever.invoke("What are some movies about dinosaurs")

/Users/avtharsewrathan/sideprojects2023/timescaleai/tsv-langchain/langchain/libs/langchain/langchain/chains/llm.py:275: UserWarning: The predict_and_parse method is deprecated, instead pass an output parser directly to LLMChain.
  warnings.warn(
``````output
query='dinosaur' filter=None limit=None

[Document(page_content='A bunch of scientists bring back dinosaurs and mayhem breaks loose', metadata={'year': 1993, 'genre': 'science fiction', 'rating': 7.7}),
 Document(page_content='A bunch of scientists bring back dinosaurs and mayhem breaks loose', metadata={'year': 1993, 'genre': 'science fiction', 'rating': 7.7}),
 Document(page_content='Toys come alive and have a blast doing so', metadata={'year': 1995, 'genre': 'animated'}),
 Document(page_content='Toys come alive and have a blast doing so', metadata={'year': 1995, 'genre': 'animated'})]

# This example only specifies a filter
retriever.invoke("I want to watch a movie rated higher than 8.5")

query=' ' filter=Comparison(comparator=<Comparator.GT: 'gt'>, attribute='rating', value=8.5) limit=None

[Document(page_content='Three men walk into the Zone, three men walk out of the Zone', metadata={'year': 1979, 'genre': 'science fiction', 'rating': 9.9, 'director': 'Andrei Tarkovsky'}),
 Document(page_content='Three men walk into the Zone, three men walk out of the Zone', metadata={'year': 1979, 'genre': 'science fiction', 'rating': 9.9, 'director': 'Andrei Tarkovsky'}),
 Document(page_content='A psychologist / detective gets lost in a series of dreams within dreams within dreams and Inception reused the idea', metadata={'year': 2006, 'rating': 8.6, 'director': 'Satoshi Kon'}),
 Document(page_content='A psychologist / detective gets lost in a series of dreams within dreams within dreams and Inception reused the idea', metadata={'year': 2006, 'rating': 8.6, 'director': 'Satoshi Kon'})]

# This example specifies a query and a filter
retriever.invoke("Has Greta Gerwig directed any movies about women")

query='women' filter=Comparison(comparator=<Comparator.EQ: 'eq'>, attribute='director', value='Greta Gerwig') limit=None

[Document(page_content='A bunch of normal-sized women are supremely wholesome and some men pine after them', metadata={'year': 2019, 'rating': 8.3, 'director': 'Greta Gerwig'}),
 Document(page_content='A bunch of normal-sized women are supremely wholesome and some men pine after them', metadata={'year': 2019, 'rating': 8.3, 'director': 'Greta Gerwig'})]

# This example specifies a composite filter
retriever.invoke("What's a highly rated (above 8.5) science fiction film?")

query=' ' filter=Operation(operator=<Operator.AND: 'and'>, arguments=[Comparison(comparator=<Comparator.GTE: 'gte'>, attribute='rating', value=8.5), Comparison(comparator=<Comparator.EQ: 'eq'>, attribute='genre', value='science fiction')]) limit=None

[Document(page_content='Three men walk into the Zone, three men walk out of the Zone', metadata={'year': 1979, 'genre': 'science fiction', 'rating': 9.9, 'director': 'Andrei Tarkovsky'}),
 Document(page_content='Three men walk into the Zone, three men walk out of the Zone', metadata={'year': 1979, 'genre': 'science fiction', 'rating': 9.9, 'director': 'Andrei Tarkovsky'})]

# This example specifies a query and composite filter
retriever.invoke(
    "What's a movie after 1990 but before 2005 that's all about toys, and preferably is animated"
)

query='toys' filter=Operation(operator=<Operator.AND: 'and'>, arguments=[Comparison(comparator=<Comparator.GT: 'gt'>, attribute='year', value=1990), Comparison(comparator=<Comparator.LT: 'lt'>, attribute='year', value=2005), Comparison(comparator=<Comparator.EQ: 'eq'>, attribute='genre', value='animated')]) limit=None

[Document(page_content='Toys come alive and have a blast doing so', metadata={'year': 1995, 'genre': 'animated'})]

过滤 k

我们也可以使用自查询检索器来指定 k：要获取的文档数量。

我们可以通过向构造函数传递 enable_limit=True 来实现这一点。

retriever = SelfQueryRetriever.from_llm(
    llm,
    vectorstore,
    document_content_description,
    metadata_field_info,
    enable_limit=True,
    verbose=True,
)

# This example specifies a query with a LIMIT value
retriever.invoke("what are two movies about dinosaurs")

query='dinosaur' filter=None limit=2

[Document(page_content='A bunch of scientists bring back dinosaurs and mayhem breaks loose', metadata={'year': 1993, 'genre': 'science fiction', 'rating': 7.7}),
 Document(page_content='A bunch of scientists bring back dinosaurs and mayhem breaks loose', metadata={'year': 1993, 'genre': 'science fiction', 'rating': 7.7})]

Timescale Vector 是什么？​

如何访问 Timescale Vector​

创建 TimescaleVector vectorstore​

创建自查询检索器​

使用 Timescale Vector 进行的自查询检索​

过滤 k​