Skip to main content
Open In ColabOpen on GitHub

TextGen

GitHub:oobabooga/text-generation-webui 一个用于运行 LLaMA、llama.cpp、GPT-J、Pythia、OPT 和 GALACTICA 等大型语言模型的 gradio web UI。

本示例将介绍如何通过 text-generation-webui API 集成使用 LangChain 与 LLM 模型进行交互。

请确保您已配置好 text-generation-webui 并安装了 LLM。推荐通过适用于您操作系统的 一键安装程序 进行安装。

安装并确认 text-generation-webui 可在 Web 界面正常工作后，请通过 Web 模型配置选项卡启用 api 选项，或在启动命令中添加运行时参数 --api。

设置 model_url 并运行示例​

model_url = "http://localhost:5000"
from langchain.chains import LLMChain
from langchain.globals import set_debug
from langchain_community.llms import TextGen
from langchain_core.prompts import PromptTemplate

set_debug(True)

template = """Question: {question}

Answer: Let's think step by step."""


prompt = PromptTemplate.from_template(template)
llm = TextGen(model_url=model_url)
llm_chain = LLMChain(prompt=prompt, llm=llm)
question = "What NFL team won the Super Bowl in the year Justin Bieber was born?"

llm_chain.run(question)

流式版本​

您应该安装 websocket-client 来使用此功能。 pip install websocket-client

model_url = "ws://localhost:5005"
from langchain.chains import LLMChain
from langchain.globals import set_debug
from langchain_community.llms import TextGen
from langchain_core.callbacks import StreamingStdOutCallbackHandler
from langchain_core.prompts import PromptTemplate

set_debug(True)

template = """Question: {question}

Answer: Let's think step by step."""


prompt = PromptTemplate.from_template(template)
llm = TextGen(
    model_url=model_url, streaming=True, callbacks=[StreamingStdOutCallbackHandler()]
)
llm_chain = LLMChain(prompt=prompt, llm=llm)
question = "What NFL team won the Super Bowl in the year Justin Bieber was born?"

llm_chain.run(question)
llm = TextGen(model_url=model_url, streaming=True)
for chunk in llm.stream("Ask 'Hi, how are you?' like a pirate:'", stop=["'", "\n"]):
    print(chunk, end="", flush=True)