结构化生成#
NIM LLM 支持通过指定 JSON 模式、正则表达式、上下文无关文法或将输出约束为某些特定选择来获得结构化输出。当 NIM 是较大管道的一部分，并且 LLM 输出需要采用特定格式时，这可能很有用。以下是一些关于如何以不同方式约束输出的示例。
JSON 模式#
您可以使用 OpenAI 模式的 nvext 扩展中的 guided_json 参数来约束输出遵循特定的 JSON 模式。这种方法在以下几种情况下特别有用
确保下游处理的一致输出格式
验证复杂数据结构
自动从非结构化文本中提取数据
提高多步骤管道的可靠性
重要提示
NVIDIA 建议您使用 guided_json 参数指定 JSON 模式，而不是设置 response_format={"type": "json_object"}。使用类型为 "json_object" 的 response_format 参数会使模型能够生成任何有效的 JSON，包括空 JSON。
基础示例：电影评论#
client = OpenAI(base_url="http://0.0.0.0:8000/v1", api_key="not-used")
json_schema = {
"type": "object",
"properties": {
"title": {
"type": "string"
},
"rating": {
"type": "number"
}
},
"required": [
"title",
"rating"
]
}
prompt = (f"Return the title and the rating based on the following movie review according to this JSON schema: {str(json_schema)}.\n"
f"Review: Inception is a really well made film. I rate it four stars out of five.")
messages = [
{"role": "user", "content": prompt},
]
response = client.chat.completions.create(
model="meta/llama-3.1-70b-instruct",
messages=messages,
extra_body={"nvext": {"guided_json": json_schema}},
stream=False
)
assistant_message = response.choices[0].message.content
print(assistant_message)
# Prints:
# {"title":"Inception", "rating":4.0}
高级示例：产品信息#
此示例演示了用于提取详细产品信息的更复杂模式
json_schema = {
"type": "object",
"properties": {
"product_name": {"type": "string"},
"price": {"type": "number"},
"features": {
"type": "array",
"items": {"type": "string"}
},
"availability": {
"type": "object",
"properties": {
"in_stock": {"type": "boolean"},
"shipping_time": {"type": "string"}
},
"required": ["in_stock", "shipping_time"]
}
},
"required": ["product_name", "price", "features", "availability"]
}
prompt = (f"Extract product information from the following description according to this JSON schema: {str(json_schema)}.\n"
f"Description: The XYZ Smartwatch is our latest offering, priced at $299.99. It features a heart rate monitor, "
f"GPS tracking, and water resistance up to 50 meters. The product is currently in stock and ships within 2-3 business days.")
messages = [
{"role": "user", "content": prompt},
]
response = client.chat.completions.create(
model="meta/llama-3.1-70b-instruct",
messages=messages,
extra_body={"nvext": {"guided_json": json_schema}},
stream=False
)
assistant_message = response.choices[0].message.content
print(assistant_message)
# Prints:
# {
# "product_name": "XYZ Smartwatch",
# "price": 299.99,
# "features": [
# "heart rate monitor",
# "GPS tracking",
# "water resistance up to 50 meters"
# ],
# "availability": {
# "in_stock": true,
# "shipping_time": "2-3 business days"
# }
# }
示例：用于活动策划的嵌套结构#
此示例展示了 JSON 模式如何处理嵌套结构，这对于复杂的数据表示非常有用
json_schema = {
"type": "object",
"properties": {
"event_name": {"type": "string"},
"date": {"type": "string", "format": "date"},
"attendees": {
"type": "array",
"items": {
"type": "object",
"properties": {
"name": {"type": "string"},
"role": {"type": "string"},
"confirmed": {"type": "boolean"}
},
"required": ["name", "role", "confirmed"]
}
},
"venue": {
"type": "object",
"properties": {
"name": {"type": "string"},
"address": {"type": "string"},
"capacity": {"type": "integer"}
},
"required": ["name", "address", "capacity"]
}
},
"required": ["event_name", "date", "attendees", "venue"]
}
prompt = (f"Create an event plan based on the following information using this JSON schema: {str(json_schema)}.\n"
f"Information: We're planning the Annual Tech Conference on 2024-09-15. John Doe (Speaker, confirmed) and Jane Smith (Organizer, confirmed) will attend. "
f"Alice Johnson (Volunteer, not confirmed yet) might join. The event will be held at Tech Center, 123 Innovation St., with a capacity of 500 people.")
messages = [
{"role": "user", "content": prompt},
]
response = client.chat.completions.create(
model="meta/llama-3.1-70b-instruct",
messages=messages,
extra_body={"nvext": {"guided_json": json_schema}},
stream=False
)
assistant_message = response.choices[0].message.content
print(assistant_message)
# Prints:
# {
# "event_name": "Annual Tech Conference",
# "date": "2024-09-15",
# "attendees": [
# {"name": "John Doe", "role": "Speaker", "confirmed": true},
# {"name": "Jane Smith", "role": "Organizer", "confirmed": true},
# {"name": "Alice Johnson", "role": "Volunteer", "confirmed": false}
# ],
# "venue": {
# "name": "Tech Center",
# "address": "123 Innovation St.",
# "capacity": 500
# }
# }
通过使用 JSON 模式，您可以确保 LLM 的输出遵循特定结构，从而更轻松地在应用程序的工作流程中处理和验证生成的数据。
正则表达式#
您可以使用 OpenAI 模式的 nvext 扩展中的 guided_regex 参数为输出格式指定正则表达式。
client = OpenAI(base_url="http://0.0.0.0:8000/v1", api_key="not-used")
regex = "[1-5]"
prompt = (f"Return just the rating based on the following movie review\n"
f"Review: This movie exceeds expectations. I rate it four stars out of five.")
messages = [
{"role": "user", "content": prompt},
]
response = client.chat.completions.create(
model="meta/llama3-8b-instruct",
messages=messages,
extra_body={"nvext": {"guided_regex": regex}},
stream=False
)
assistant_message = response.choices[0].message.content
print(assistant_message)
# Prints:
# 4
选项#
您可以使用 OpenAI 模式的 nvext 扩展中的 guided_choice 参数为输出指定选项列表。
client = OpenAI(base_url="http://0.0.0.0:8000/v1", api_key="not-used")
choices = ["Good", "Bad", "Neutral"]
prompt = (f"Return the sentiment based on the following movie review. It should be one of {choices}\n"
f"Review: This movie exceeds expectations. I rate it four stars out of five.")
messages = [
{"role": "user", "content": prompt},
]
response = client.chat.completions.create(
model="meta/llama3-8b-instruct",
messages=messages,
extra_body={"nvext": {"guided_choice": choices}},
stream=False
)
assistant_message = response.choices[0].message.content
print(assistant_message)
# Prints:
# Good
上下文无关文法#
您可以使用 OpenAI 模式的 nvext 扩展中的 guided_grammar 参数以 EBNF 格式指定上下文无关文法。
client = OpenAI(base_url="http://0.0.0.0:8000/v1", api_key="not-used")
grammar = """
?start: "The movie name is rated " rating " stars."
?rating: /[1-5]/
"""
prompt = (f"Summarize the following movie review:\n"
f"Review: This movie exceeds expectations. I rate it four stars out of five.")
messages = [
{"role": "user", "content": prompt},
]
response = client.chat.completions.create(
model="meta/llama3-8b-instruct",
messages=messages,
extra_body={"nvext": {"guided_grammar": grammar}},
stream=False
)
completion = response.choices[0].message.content
print(completion)
# Prints:
# The movie name is rated 4 stars.