Skip to main content
Open In ColabOpen on GitHub

Titan Takeoff

TitanML 通过我们的训练、压缩和推理优化平台，帮助企业构建和部署更好、更小、更便宜、更快的 NLP 模型。

我们的推理服务器 Titan Takeoff 能够通过单个命令在您的本地硬件上部署 LLM。支持大多数生成模型架构，例如 Falcon、Llama 2、GPT2、T5 等。如果您在使用特定模型时遇到问题，请通过 hello@titanml.co 告知我们。

示例用法​

以下是一些有助于您开始使用 Titan Takeoff Server 的示例。在运行这些命令之前，您需要确保 Takeoff Server 已在后台启动。有关更多信息，请参阅启动 Takeoff 的文档页面。

import time

# Note importing TitanTakeoffPro instead of TitanTakeoff will work as well both use same object under the hood
from langchain_community.llms import TitanTakeoff
from langchain_core.callbacks import CallbackManager, StreamingStdOutCallbackHandler
from langchain_core.prompts import PromptTemplate

示例 1​

基本用法，假设 Takeoff 在您的机器上运行，并使用其默认端口（即 localhost:3000）。

llm = TitanTakeoff()
output = llm.invoke("What is the weather in London in August?")
print(output)

示例 2​

指定端口和其他生成参数

llm = TitanTakeoff(port=3000)
# A comprehensive list of parameters can be found at https://docs.titanml.co/docs/next/apis/Takeoff%20inference_REST_API/generate#request
output = llm.invoke(
    "What is the largest rainforest in the world?",
    consumer_group="primary",
    min_new_tokens=128,
    max_new_tokens=512,
    no_repeat_ngram_size=2,
    sampling_topk=1,
    sampling_topp=1.0,
    sampling_temperature=1.0,
    repetition_penalty=1.0,
    regex_string="",
    json_schema=None,
)
print(output)

示例 3​

使用 generate 处理多个输入

llm = TitanTakeoff()
rich_output = llm.generate(["What is Deep Learning?", "What is Machine Learning?"])
print(rich_output.generations)

示例 4​

流式输出

llm = TitanTakeoff(
    streaming=True, callback_manager=CallbackManager([StreamingStdOutCallbackHandler()])
)
prompt = "What is the capital of France?"
output = llm.invoke(prompt)
print(output)

示例 5​

使用 LCEL

llm = TitanTakeoff()
prompt = PromptTemplate.from_template("Tell me about {topic}")
chain = prompt | llm
output = chain.invoke({"topic": "the universe"})
print(output)

示例 6​

使用 TitanTakeoff Python 包装器启动读取器。如果你在首次启动 Takeoff 时没有创建任何读取器，或者想添加另一个读取器，可以在初始化 TitanTakeoff 对象时进行操作。只需将你想启动的模型配置列表作为 models 参数传递即可。

# Model config for the llama model, where you can specify the following parameters:
#   model_name (str): The name of the model to use
#   device: (str): The device to use for inference, cuda or cpu
#   consumer_group (str): The consumer group to place the reader into
#   tensor_parallel (Optional[int]): The number of gpus you would like your model to be split across
#   max_seq_length (int): The maximum sequence length to use for inference, defaults to 512
#   max_batch_size (int_: The max batch size for continuous batching of requests
llama_model = {
    "model_name": "TheBloke/Llama-2-7b-Chat-AWQ",
    "device": "cuda",
    "consumer_group": "llama",
}
llm = TitanTakeoff(models=[llama_model])

# The model needs time to spin up, length of time need will depend on the size of model and your network connection speed
time.sleep(60)

prompt = "What is the capital of France?"
output = llm.invoke(prompt, consumer_group="llama")
print(output)