Confident

DeepEval 包用于 LLM 的单元测试。使用 Confident，每个人都可以通过更快的迭代来构建健壮的语言模型，并且可以同时进行单元测试和集成测试。我们为迭代的每个步骤提供支持，从合成数据创建到测试。

在本指南中，我们将演示如何测试和衡量 LLM 的性能。我们将展示如何使用我们的回调来衡量性能，以及如何定义自己的指标并将其记录到我们的仪表板中。

DeepEval 还提供：

如何生成合成数据
如何衡量性能
用于随着时间推移监控和审查结果的仪表板

安装与设置

%pip install --upgrade --quiet  langchain langchain-openai langchain-community deepeval langchain-chroma

获取 API 凭证

要获取 DeepEval API 凭证，请按照以下步骤操作：

前往 https://app.confident-ai.com
点击“Organization”
复制 API Key。

登录时，系统还会要求你设置 implementation 名称。此 implementation 名称用于描述实现的类型。（可以是你项目的名称。我们建议将其命名得更具描述性。）

!deepeval login

设置 DeepEval

默认情况下，您可以使用 DeepEvalCallbackHandler 来设置您想要跟踪的指标。但是，目前它对指标的支持有限（即将添加更多）。它目前支持：

from deepeval.metrics.answer_relevancy import AnswerRelevancy

# Here we want to make sure the answer is minimally relevant
answer_relevancy_metric = AnswerRelevancy(minimum_score=0.5)

入门

要使用 DeepEvalCallbackHandler，我们需要 implementation_name。

from langchain_community.callbacks.confident_callback import DeepEvalCallbackHandler

deepeval_callback = DeepEvalCallbackHandler(
    implementation_name="langchainQuickstart", metrics=[answer_relevancy_metric]
)

API Reference:DeepEvalCallbackHandler

场景 1：输入到 LLM

然后，你可以将其输入到你的带有 OpenAI 的 LLM 中。

from langchain_openai import OpenAI

llm = OpenAI(
    temperature=0,
    callbacks=[deepeval_callback],
    verbose=True,
    openai_api_key="<YOUR_API_KEY>",
)
output = llm.generate(
    [
        "What is the best evaluation tool out there? (no bias at all)",
    ]
)

API Reference:OpenAI

LLMResult(generations=[[Generation(text='\n\nQ: What did the fish say when he hit the wall? \nA: Dam.', generation_info={'finish_reason': 'stop', 'logprobs': None})], [Generation(text='\n\nThe Moon \n\nThe moon is high in the midnight sky,\nSparkling like a star above.\nThe night so peaceful, so serene,\nFilling up the air with love.\n\nEver changing and renewing,\nA never-ending light of grace.\nThe moon remains a constant view,\nA reminder of life’s gentle pace.\n\nThrough time and space it guides us on,\nA never-fading beacon of hope.\nThe moon shines down on us all,\nAs it continues to rise and elope.', generation_info={'finish_reason': 'stop', 'logprobs': None})], [Generation(text='\n\nQ. What did one magnet say to the other magnet?\nA. "I find you very attractive!"', generation_info={'finish_reason': 'stop', 'logprobs': None})], [Generation(text="\n\nThe world is charged with the grandeur of God.\nIt will flame out, like shining from shook foil;\nIt gathers to a greatness, like the ooze of oil\nCrushed. Why do men then now not reck his rod?\n\nGenerations have trod, have trod, have trod;\nAnd all is seared with trade; bleared, smeared with toil;\nAnd wears man's smudge and shares man's smell: the soil\nIs bare now, nor can foot feel, being shod.\n\nAnd for all this, nature is never spent;\nThere lives the dearest freshness deep down things;\nAnd though the last lights off the black West went\nOh, morning, at the brown brink eastward, springs —\n\nBecause the Holy Ghost over the bent\nWorld broods with warm breast and with ah! bright wings.\n\n~Gerard Manley Hopkins", generation_info={'finish_reason': 'stop', 'logprobs': None})], [Generation(text='\n\nQ: What did one ocean say to the other ocean?\nA: Nothing, they just waved.', generation_info={'finish_reason': 'stop', 'logprobs': None})], [Generation(text="\n\nA poem for you\n\nOn a field of green\n\nThe sky so blue\n\nA gentle breeze, the sun above\n\nA beautiful world, for us to love\n\nLife is a journey, full of surprise\n\nFull of joy and full of surprise\n\nBe brave and take small steps\n\nThe future will be revealed with depth\n\nIn the morning, when dawn arrives\n\nA fresh start, no reason to hide\n\nSomewhere down the road, there's a heart that beats\n\nBelieve in yourself, you'll always succeed.", generation_info={'finish_reason': 'stop', 'logprobs': None})]], llm_output={'token_usage': {'completion_tokens': 504, 'total_tokens': 528, 'prompt_tokens': 24}, 'model_name': 'text-davinci-003'})

然后，你可以通过调用 is_successful() 方法来检查该指标的成功状态。

answer_relevancy_metric.is_successful()
# returns True/False

运行后，您应该能在下方看到我们的仪表板。

仪表板

场景 2：在链中跟踪 LLM（无回调）

要在链中跟踪 LLM 但不使用回调，您可以在末尾插入。

我们可以从定义一个简单的链开始，如下所示。

import requests
from langchain.chains import RetrievalQA
from langchain_chroma import Chroma
from langchain_community.document_loaders import TextLoader
from langchain_openai import OpenAI, OpenAIEmbeddings
from langchain_text_splitters import CharacterTextSplitter

text_file_url = "https://raw.githubusercontent.com/hwchase17/chat-your-data/master/state_of_the_union.txt"

openai_api_key = "sk-XXX"

with open("state_of_the_union.txt", "w") as f:
    response = requests.get(text_file_url)
    f.write(response.text)

loader = TextLoader("state_of_the_union.txt")
documents = loader.load()
text_splitter = CharacterTextSplitter(chunk_size=1000, chunk_overlap=0)
texts = text_splitter.split_documents(documents)

embeddings = OpenAIEmbeddings(openai_api_key=openai_api_key)
docsearch = Chroma.from_documents(texts, embeddings)

qa = RetrievalQA.from_chain_type(
    llm=OpenAI(openai_api_key=openai_api_key),
    chain_type="stuff",
    retriever=docsearch.as_retriever(),
)

# Providing a new question-answering pipeline
query = "Who is the president?"
result = qa.run(query)

定义链之后，您可以手动检查答案相似度。

answer_relevancy_metric.measure(result, query)
answer_relevancy_metric.is_successful()

下一步？

你可以在此处创建自己的自定义指标。

DeepEval 还提供其他功能，例如能够自动创建单元测试、测试幻觉。

如果你感兴趣，可以在此处查看我们的 Github 仓库。我们欢迎任何关于如何改进 LLM 性能的 PR 和讨论。

安装与设置​

获取 API 凭证​

设置 DeepEval​

入门​

场景 1：输入到 LLM​

场景 2：在链中跟踪 LLM（无回调）​

下一步？​