在這個教學中,你會打造一個完全在裝置上運行的互動聊天助理。 助理會維持多次對話的上下文,因此會記得你之前討論的內容。 你使用 Foundry Local SDK 來選擇模型、定義系統提示,並逐字串流輸出回應。
在本教學課程中,您將瞭解如何:
- 建立專案並安裝 Foundry 本地 SDK
- 瀏覽型號目錄並選擇一個型號
- 定義一個系統提示詞來塑造助理行為
- 實現多輪對話及訊息歷史功能
- 串流回應,帶來即時回應的體驗
- 對話結束後清理資源
先決條件
- 一台至少有 8GB 記憶體的 Windows、macOS 或 Linux 電腦。
樣本庫
你可以在 Foundry 範例 GitHub 倉庫找到這篇文章的完整範例程式碼。 要複製儲存庫並導向範例,請使用:
git clone https://github.com/microsoft-foundry/foundry-samples.git
cd foundry-samples/samples/csharp/foundry-local/tutorial-chat-assistant
安裝套件
dotnet add package Microsoft.AI.Foundry.Local
dotnet add package OpenAI
GitHub 倉庫裡的 C# 範例都是預先設定好的專案。 如果你是從零開始建立,建議閱讀 Foundry Local SDK 參考資料 ,了解更多如何用 Foundry Local 設定 C# 專案的細節。
瀏覽目錄並選擇型號
Foundry Local SDK 提供一個模型目錄,列出所有可用模型。 在這個步驟中,你會初始化 SDK,並為你的聊天助理選擇一個模型。
打開
Program.cs並以以下程式碼替換其內容,以初始化 SDK 並選擇模型:CancellationToken ct = CancellationToken.None; var config = new Configuration { AppName = "foundry_local_samples", LogLevel = Microsoft.AI.Foundry.Local.LogLevel.Information }; using var loggerFactory = LoggerFactory.Create(builder => { builder.SetMinimumLevel(Microsoft.Extensions.Logging.LogLevel.Information); }); var logger = loggerFactory.CreateLogger<Program>(); // Initialize the singleton instance await FoundryLocalManager.CreateAsync(config, logger); var mgr = FoundryLocalManager.Instance; // Download and register all execution providers. var currentEp = ""; await mgr.DownloadAndRegisterEpsAsync((epName, percent) => { if (epName != currentEp) { if (currentEp != "") Console.WriteLine(); currentEp = epName; } Console.Write($"\r {epName.PadRight(30)} {percent,6:F1}%"); }); if (currentEp != "") Console.WriteLine(); // Select and load a model from the catalog var catalog = await mgr.GetCatalogAsync(); var model = await catalog.GetModelAsync("qwen2.5-0.5b") ?? throw new Exception("Model not found"); await model.DownloadAsync(progress => { Console.Write($"\rDownloading model: {progress:F2}%"); if (progress >= 100f) Console.WriteLine(); }); await model.LoadAsync(); Console.WriteLine("Model loaded and ready."); // Create one stateful native chat session for the conversation. using var chatSession = new ChatSession(model); chatSession.SetStreaming(true);此
GetModelAsync方法接受模型別名,這是一種短且友善的名稱,對應到目錄中的特定模型。 這個DownloadAsync方法會將模型權重取到本地快取,並LoadAsync讓模型準備好進行推論。
定義系統提示
系統提示詞會設定助理的個性與行為。 這是對話紀錄中的第一則訊息,模型在整個對話過程中都會提及它。
新增系統提示來塑造助理的回應方式:
// Start the conversation with a system prompt
var systemPrompt = "You are a helpful, friendly assistant. Keep your responses " +
"concise and conversational. If you don't know something, say so.";
小提示
嘗試不同的系統提示來改變助理的行為。 例如,你可以指示它以海盜、老師或領域專家的身份回應。
實作多回合交談
聊天助理需要在多個交換中維持上下文。 你可以透過保留一份包含所有訊息(系統、使用者和助理)的清單,並在每個請求時傳送完整清單來達成這個目標。 模型利用這些歷史來產生具上下文相關的回應。
新增一個對話循環:
- 讀取主機的使用者輸入。
- 將使用者訊息附加到歷史紀錄中。
- 將完整的歷史資料傳送給模型。
- 將助手的回應附加到下一回合的上下文中。
while (true)
{
Console.Write("You: ");
var userInput = Console.ReadLine();
if (string.IsNullOrWhiteSpace(userInput) ||
userInput.Equals("quit", StringComparison.OrdinalIgnoreCase) ||
userInput.Equals("exit", StringComparison.OrdinalIgnoreCase))
{
break;
}
using var request = new Request();
if (chatSession.TurnCount == 0)
{
request.AddItem(MessageItem.System(systemPrompt));
}
request.AddItem(MessageItem.User(userInput));
// Stream the response token by token
Console.Write("Assistant: ");
await using var streamingResponse = chatSession.ProcessStreamingRequestAsync(request, ct);
await foreach (var item in streamingResponse)
{
using (item)
{
if (item is TextItem { Type: TextItemType.Default } text)
{
Console.Write(text.Text);
Console.Out.Flush();
}
}
}
using var finalResponse = await streamingResponse.FinalResponse;
Console.WriteLine("\n");
}
每次呼叫 CompleteChatAsync 都會收到完整的訊息歷史。 這就是模型「記住」前幾回合的方式——它不會在呼叫之間儲存狀態。
新增串流回應
串流會在每個權杖產生時印出,使助理更能迅速給予回覆。 將CompleteChatAsync呼叫替換為CompleteChatStreamingAsync以流式傳輸每個回應標記。
更新對話循環以使用串流:
// Stream the response token by token
Console.Write("Assistant: ");
await using var streamingResponse = chatSession.ProcessStreamingRequestAsync(request, ct);
await foreach (var item in streamingResponse)
{
using (item)
{
if (item is TextItem { Type: TextItemType.Default } text)
{
Console.Write(text.Text);
Console.Out.Flush();
}
}
}
using var finalResponse = await streamingResponse.FinalResponse;
Console.WriteLine("\n");
串流版本會累積完整回應,這樣直播結束後就能加入對話紀錄。
完整程式碼
將 Program.cs 的內容替換為以下完整程式碼:
using Microsoft.AI.Foundry.Local;
using Microsoft.Extensions.Logging;
CancellationToken ct = CancellationToken.None;
var config = new Configuration
{
AppName = "foundry_local_samples",
LogLevel = Microsoft.AI.Foundry.Local.LogLevel.Information
};
using var loggerFactory = LoggerFactory.Create(builder =>
{
builder.SetMinimumLevel(Microsoft.Extensions.Logging.LogLevel.Information);
});
var logger = loggerFactory.CreateLogger<Program>();
// Initialize the singleton instance
await FoundryLocalManager.CreateAsync(config, logger);
var mgr = FoundryLocalManager.Instance;
// Download and register all execution providers.
var currentEp = "";
await mgr.DownloadAndRegisterEpsAsync((epName, percent) =>
{
if (epName != currentEp)
{
if (currentEp != "") Console.WriteLine();
currentEp = epName;
}
Console.Write($"\r {epName.PadRight(30)} {percent,6:F1}%");
});
if (currentEp != "") Console.WriteLine();
// Select and load a model from the catalog
var catalog = await mgr.GetCatalogAsync();
var model = await catalog.GetModelAsync("qwen2.5-0.5b")
?? throw new Exception("Model not found");
await model.DownloadAsync(progress =>
{
Console.Write($"\rDownloading model: {progress:F2}%");
if (progress >= 100f) Console.WriteLine();
});
await model.LoadAsync();
Console.WriteLine("Model loaded and ready.");
// Create one stateful native chat session for the conversation.
using var chatSession = new ChatSession(model);
chatSession.SetStreaming(true);
// Start the conversation with a system prompt
var systemPrompt = "You are a helpful, friendly assistant. Keep your responses " +
"concise and conversational. If you don't know something, say so.";
Console.WriteLine("\nChat assistant ready! Type 'quit' to exit.\n");
while (true)
{
Console.Write("You: ");
var userInput = Console.ReadLine();
if (string.IsNullOrWhiteSpace(userInput) ||
userInput.Equals("quit", StringComparison.OrdinalIgnoreCase) ||
userInput.Equals("exit", StringComparison.OrdinalIgnoreCase))
{
break;
}
using var request = new Request();
if (chatSession.TurnCount == 0)
{
request.AddItem(MessageItem.System(systemPrompt));
}
request.AddItem(MessageItem.User(userInput));
// Stream the response token by token
Console.Write("Assistant: ");
await using var streamingResponse = chatSession.ProcessStreamingRequestAsync(request, ct);
await foreach (var item in streamingResponse)
{
using (item)
{
if (item is TextItem { Type: TextItemType.Default } text)
{
Console.Write(text.Text);
Console.Out.Flush();
}
}
}
using var finalResponse = await streamingResponse.FinalResponse;
Console.WriteLine("\n");
}
// Clean up - unload the model
chatSession.Dispose();
await model.UnloadAsync();
Console.WriteLine("Model unloaded. Goodbye!");
運行聊天助理:
dotnet run
你會看到類似的輸出:
Downloading model: 100.00%
Model loaded and ready.
Chat assistant ready! Type 'quit' to exit.
You: What is photosynthesis?
Assistant: Photosynthesis is the process plants use to convert sunlight, water, and carbon
dioxide into glucose and oxygen. It mainly happens in the leaves, inside structures
called chloroplasts.
You: Why is it important for other living things?
Assistant: It's essential because photosynthesis produces the oxygen that most living things
breathe. It also forms the base of the food chain — animals eat plants or eat other
animals that depend on plants for energy.
You: quit
Model unloaded. Goodbye!
注意助理如何記住之前回合的上下文——當你問「為什麼對其他生物重要?」時,它知道你仍在談論光合作用。
樣本庫
本文的完整範例程式碼可在 Foundry-samples GitHub 倉庫取得。 要複製儲存庫並導向範例,請使用:
git clone https://github.com/microsoft-foundry/foundry-samples.git
cd foundry-samples/samples/javascript/foundry-local/tutorial-chat-assistant
安裝套件
npm install foundry-local-sdk openai
瀏覽目錄並選擇型號
Foundry Local SDK 提供一個模型目錄,列出所有可用模型。 在這個步驟中,你會初始化 SDK,並為你的聊天助理選擇一個模型。
建立一個名為
index.js. 的檔案。請加入以下程式碼來初始化 SDK 並選擇模型:
// Initialize the Foundry Local SDK const manager = FoundryLocalManager.create({ appName: 'foundry_local_samples', logLevel: 'info' }); // Download and register all execution providers. let currentEp = ''; await manager.downloadAndRegisterEps((epName, percent) => { if (epName !== currentEp) { if (currentEp !== '') process.stdout.write('\n'); currentEp = epName; } process.stdout.write(`\r ${epName.padEnd(30)} ${percent.toFixed(1).padStart(5)}%`); }); if (currentEp !== '') process.stdout.write('\n'); // Select and load a model from the catalog const model = await manager.catalog.getModel('qwen2.5-0.5b'); await model.download((progress) => { process.stdout.write(`\rDownloading model: ${progress.toFixed(2)}%`); }); console.log('\nModel downloaded.'); await model.load(); console.log('Model loaded and ready.'); // Create a stateful native chat session const chatSession = new ChatSession(model); chatSession.setOptions({ search: { temperature: 0.7, maxOutputTokens: 512 } });此
getModel方法接受模型別名,這是一種短且友善的名稱,對應到目錄中的特定模型。 這個download方法會將模型權重取到本地快取,並load讓模型準備好進行推論。
定義系統提示
系統提示詞會設定助理的個性與行為。 這是對話紀錄中的第一則訊息,模型在整個對話過程中都會提及它。
新增系統提示來塑造助理的回應方式:
// Start the conversation with a system prompt
const systemPrompt =
'You are a helpful, friendly assistant. Keep your responses ' +
'concise and conversational. If you don\'t know something, say so.';
let firstTurn = true;
小提示
嘗試不同的系統提示來改變助理的行為。 例如,你可以指示它以海盜、老師或領域專家的身份回應。
實作多回合交談
聊天助理需要在多個交換中維持上下文。 你可以透過保留一份包含所有訊息(系統、使用者和助理)的清單,並在每個請求時傳送完整清單來達成這個目標。 模型利用這些歷史來產生具上下文相關的回應。
新增一個對話循環:
- 讀取主機的使用者輸入。
- 將使用者訊息附加到歷史紀錄中。
- 將完整的歷史資料傳送給模型。
- 將助手的回應附加到下一回合的上下文中。
while (true) {
const userInput = await askQuestion('You: ');
if (userInput.trim().toLowerCase() === 'quit' ||
userInput.trim().toLowerCase() === 'exit') {
break;
}
const request = new Request();
if (firstTurn) {
request.addItem(Item.systemMessage(systemPrompt));
firstTurn = false;
}
request.addItem(Item.userMessage(userInput));
// Stream the response token by token
process.stdout.write('Assistant: ');
const stream = chatSession.processStreamingRequest(request);
for await (const item of stream) {
if (item.type === 'text' && item.textType === 'default') {
process.stdout.write(item.text);
}
}
await stream.response;
console.log('\n');
}
每次呼叫 completeChat 都會收到完整的訊息歷史。 這就是模型「記住」前幾回合的方式——它不會在呼叫之間儲存狀態。
新增串流回應
串流會在每個權杖產生時印出,使助理更能迅速給予回覆。 將completeChat呼叫替換為completeStreamingChat以流式傳輸每個回應標記。
更新對話循環以使用串流:
// Stream the response token by token
process.stdout.write('Assistant: ');
const stream = chatSession.processStreamingRequest(request);
for await (const item of stream) {
if (item.type === 'text' && item.textType === 'default') {
process.stdout.write(item.text);
}
}
await stream.response;
console.log('\n');
串流版本會累積完整回應,這樣直播結束後就能加入對話紀錄。
完整程式碼
建立一個名為 index.js S 的檔案,並加入以下完整程式碼:
import {
ChatSession,
FoundryLocalManager,
Item,
Request
} from 'foundry-local-sdk';
import * as readline from 'readline';
// Initialize the Foundry Local SDK
const manager = FoundryLocalManager.create({
appName: 'foundry_local_samples',
logLevel: 'info'
});
// Download and register all execution providers.
let currentEp = '';
await manager.downloadAndRegisterEps((epName, percent) => {
if (epName !== currentEp) {
if (currentEp !== '') process.stdout.write('\n');
currentEp = epName;
}
process.stdout.write(`\r ${epName.padEnd(30)} ${percent.toFixed(1).padStart(5)}%`);
});
if (currentEp !== '') process.stdout.write('\n');
// Select and load a model from the catalog
const model = await manager.catalog.getModel('qwen2.5-0.5b');
await model.download((progress) => {
process.stdout.write(`\rDownloading model: ${progress.toFixed(2)}%`);
});
console.log('\nModel downloaded.');
await model.load();
console.log('Model loaded and ready.');
// Create a stateful native chat session
const chatSession = new ChatSession(model);
chatSession.setOptions({
search: { temperature: 0.7, maxOutputTokens: 512 }
});
// Start the conversation with a system prompt
const systemPrompt =
'You are a helpful, friendly assistant. Keep your responses ' +
'concise and conversational. If you don\'t know something, say so.';
let firstTurn = true;
// Set up readline for console input
const rl = readline.createInterface({
input: process.stdin,
output: process.stdout
});
const askQuestion = (prompt) => new Promise((resolve) => {
if (rl.closed) return resolve('quit');
const onClose = () => resolve('quit');
rl.once('close', onClose);
try {
rl.question(prompt, (answer) => {
rl.off('close', onClose);
resolve(answer);
});
} catch {
rl.off('close', onClose);
resolve('quit');
}
});
console.log('\nChat assistant ready! Type \'quit\' to exit.\n');
while (true) {
const userInput = await askQuestion('You: ');
if (userInput.trim().toLowerCase() === 'quit' ||
userInput.trim().toLowerCase() === 'exit') {
break;
}
const request = new Request();
if (firstTurn) {
request.addItem(Item.systemMessage(systemPrompt));
firstTurn = false;
}
request.addItem(Item.userMessage(userInput));
// Stream the response token by token
process.stdout.write('Assistant: ');
const stream = chatSession.processStreamingRequest(request);
for await (const item of stream) {
if (item.type === 'text' && item.textType === 'default') {
process.stdout.write(item.text);
}
}
await stream.response;
console.log('\n');
}
// Clean up - unload the model
chatSession.dispose();
await model.unload();
console.log('Model unloaded. Goodbye!');
rl.close();
運行聊天助理:
node index.js
你會看到類似的輸出:
Downloading model: 100.00%
Model downloaded.
Model loaded and ready.
Chat assistant ready! Type 'quit' to exit.
You: What is photosynthesis?
Assistant: Photosynthesis is the process plants use to convert sunlight, water, and carbon
dioxide into glucose and oxygen. It mainly happens in the leaves, inside structures
called chloroplasts.
You: Why is it important for other living things?
Assistant: It's essential because photosynthesis produces the oxygen that most living things
breathe. It also forms the base of the food chain — animals eat plants or eat other
animals that depend on plants for energy.
You: quit
Model unloaded. Goodbye!
注意助理如何記住之前回合的上下文——當你問「為什麼對其他生物重要?」時,它知道你仍在談論光合作用。
樣本庫
本文的完整範例程式碼可在 Foundry-samples GitHub 倉庫取得。 要複製儲存庫並導向範例,請使用:
git clone https://github.com/microsoft-foundry/foundry-samples.git
cd foundry-samples/samples/python/foundry-local/tutorial-chat-assistant
安裝套件
pip install foundry-local-sdk openai
瀏覽目錄並選擇型號
Foundry Local SDK 提供一個模型目錄,列出所有可用模型。 在這個步驟中,你會初始化 SDK,並為你的聊天助理選擇一個模型。
建立一個名為
main.py. 的檔案。請加入以下程式碼來初始化 SDK 並選擇模型:
# Initialize the Foundry Local SDK config = Configuration(app_name="foundry_local_samples") FoundryLocalManager.initialize(config) manager = FoundryLocalManager.instance # Discover and register execution providers that are not already available. missing_eps = [ep.name for ep in manager.discover_eps() if not ep.is_registered] if missing_eps: manager.download_and_register_eps(missing_eps) # Select and load a model from the catalog. model = manager.catalog.get_model("qwen2.5-0.5b") if model is None: raise RuntimeError("Model 'qwen2.5-0.5b' was not found in the catalog.") if not model.is_cached: model.download( lambda progress: print( f"\rDownloading model: {progress:.2f}%", end="", flush=True ) ) print() model.load() print("Model loaded and ready.")此
get_model方法接受模型別名,這是一種短且友善的名稱,對應到目錄中的特定模型。 這個download方法會將模型權重取到本地快取,並load讓模型準備好進行推論。
定義系統提示
系統提示詞會設定助理的個性與行為。 這是對話紀錄中的第一則訊息,模型在整個對話過程中都會提及它。
新增系統提示來塑造助理的回應方式:
# Start the conversation with a system prompt.
system_prompt = (
"You are a helpful, friendly assistant. Keep your responses "
"concise and conversational. If you don't know something, say so."
)
小提示
嘗試不同的系統提示來改變助理的行為。 例如,你可以指示它以海盜、老師或領域專家的身份回應。
實作多回合交談
聊天助理需要在多個交換中維持上下文。 你可以透過保留一份包含所有訊息(系統、使用者和助理)的清單,並在每個請求時傳送完整清單來達成這個目標。 模型利用這些歷史來產生具上下文相關的回應。
新增一個對話循環:
- 讀取主機的使用者輸入。
- 將使用者訊息附加到歷史紀錄中。
- 將完整的歷史資料傳送給模型。
- 將助手的回應附加到下一回合的上下文中。
while True:
user_input = input("You: ")
if user_input.strip().lower() in ("quit", "exit"):
break
# The stateful session retains earlier turns automatically.
with Request() as request:
if first_turn:
request.add_item(MessageItem.system(system_prompt))
first_turn = False
request.add_item(MessageItem.user(user_input))
print("Assistant: ", end="", flush=True)
with session.process_streaming_request(request) as stream:
for item in stream:
if (
isinstance(item, TextItem)
and item.type == TextItemType.DEFAULT
):
print(item.text, end="", flush=True)
print("\n")
每次呼叫 complete_chat 都會收到完整的訊息歷史。 這就是模型「記住」前幾回合的方式——它不會在呼叫之間儲存狀態。
新增串流回應
串流會在每個權杖產生時印出,使助理更能迅速給予回覆。 將complete_chat呼叫替換為complete_streaming_chat以流式傳輸每個回應標記。
更新對話循環以使用串流:
# The stateful session retains earlier turns automatically.
with Request() as request:
if first_turn:
request.add_item(MessageItem.system(system_prompt))
first_turn = False
request.add_item(MessageItem.user(user_input))
print("Assistant: ", end="", flush=True)
with session.process_streaming_request(request) as stream:
for item in stream:
if (
isinstance(item, TextItem)
and item.type == TextItemType.DEFAULT
):
print(item.text, end="", flush=True)
print("\n")
串流版本會累積完整回應,這樣直播結束後就能加入對話紀錄。
完整程式碼
建立一個名為 main.py S 的檔案,並加入以下完整程式碼:
from foundry_local_sdk import (
ChatSession,
Configuration,
FoundryLocalManager,
MessageItem,
Request,
RequestOptions,
SearchOptions,
TextItem,
TextItemType,
)
def main():
# Initialize the Foundry Local SDK
config = Configuration(app_name="foundry_local_samples")
FoundryLocalManager.initialize(config)
manager = FoundryLocalManager.instance
# Discover and register execution providers that are not already available.
missing_eps = [ep.name for ep in manager.discover_eps() if not ep.is_registered]
if missing_eps:
manager.download_and_register_eps(missing_eps)
# Select and load a model from the catalog.
model = manager.catalog.get_model("qwen2.5-0.5b")
if model is None:
raise RuntimeError("Model 'qwen2.5-0.5b' was not found in the catalog.")
if not model.is_cached:
model.download(
lambda progress: print(
f"\rDownloading model: {progress:.2f}%", end="", flush=True
)
)
print()
model.load()
print("Model loaded and ready.")
try:
# Start the conversation with a system prompt.
system_prompt = (
"You are a helpful, friendly assistant. Keep your responses "
"concise and conversational. If you don't know something, say so."
)
print("\nChat assistant ready! Type 'quit' to exit.\n")
with ChatSession(model) as session:
session.set_options(
RequestOptions(
search=SearchOptions(temperature=0.7, max_output_tokens=512)
)
)
session.set_streaming(True)
first_turn = True
while True:
user_input = input("You: ")
if user_input.strip().lower() in ("quit", "exit"):
break
# The stateful session retains earlier turns automatically.
with Request() as request:
if first_turn:
request.add_item(MessageItem.system(system_prompt))
first_turn = False
request.add_item(MessageItem.user(user_input))
print("Assistant: ", end="", flush=True)
with session.process_streaming_request(request) as stream:
for item in stream:
if (
isinstance(item, TextItem)
and item.type == TextItemType.DEFAULT
):
print(item.text, end="", flush=True)
print("\n")
finally:
# Clean up - unload the model
if model.is_loaded:
model.unload()
manager.close()
print("Model unloaded. Goodbye!")
if __name__ == "__main__":
main()
運行聊天助理:
python main.py
你會看到類似的輸出:
Downloading model: 100.00%
Model loaded and ready.
Chat assistant ready! Type 'quit' to exit.
You: What is photosynthesis?
Assistant: Photosynthesis is the process plants use to convert sunlight, water, and carbon
dioxide into glucose and oxygen. It mainly happens in the leaves, inside structures
called chloroplasts.
You: Why is it important for other living things?
Assistant: It's essential because photosynthesis produces the oxygen that most living things
breathe. It also forms the base of the food chain — animals eat plants or eat other
animals that depend on plants for energy.
You: quit
Model unloaded. Goodbye!
注意助理如何記住之前回合的上下文——當你問「為什麼對其他生物重要?」時,它知道你仍在談論光合作用。
樣本庫
本文的完整範例程式碼可在 Foundry-samples GitHub 倉庫取得。 要複製儲存庫並導向範例,請使用:
git clone https://github.com/microsoft-foundry/foundry-samples.git
cd foundry-samples/samples/rust/foundry-local/tutorial-chat-assistant
安裝套件
cargo add foundry-local-sdk
cargo add tokio --features full
cargo add tokio-stream anyhow
瀏覽目錄並選擇型號
Foundry Local SDK 提供一個模型目錄,列出所有可用模型。 在這個步驟中,你會初始化 SDK,並為你的聊天助理選擇一個模型。
打開
src/main.rs並以以下程式碼替換其內容,以初始化 SDK 並選擇模型:// Initialize the Foundry Local SDK let manager = FoundryLocalManager::create(FoundryLocalConfig::new("foundry_local_samples"))?; manager .download_and_register_eps_with_progress(None, { let mut current_ep = String::new(); move |ep_name: &str, percent: f64| { if ep_name != current_ep { if !current_ep.is_empty() { println!(); } current_ep = ep_name.to_string(); } print!("\r {ep_name:<30} {percent:5.1}%"); io::stdout().flush().ok(); } }) .await?; println!(); let model = manager.catalog().get_model("qwen2.5-0.5b").await?; if !model.is_cached().await? { println!("Downloading model..."); model .download(Some(|progress: f64| { print!("\r {progress:.1}%"); io::stdout().flush().ok(); })) .await?; println!(); } model.load().await?; let session = ChatSession::new(&model).await?; session .set_options(RequestOptions { search: SearchOptions { temperature: Some(0.7), max_output_tokens: Some(512), ..Default::default() }, ..Default::default() }) .await?; println!("Model loaded and ready.");此
get_model方法接受模型別名,這是一種短且友善的名稱,對應到目錄中的特定模型。 這個download方法會將模型權重取到本地快取,並load讓模型準備好進行推論。
定義系統提示
系統提示詞會設定助理的個性與行為。 這是對話紀錄中的第一則訊息,模型在整個對話過程中都會提及它。
新增系統提示來塑造助理的回應方式:
let system_prompt = "You are a helpful, friendly assistant. Keep your responses concise and \
conversational. If you don't know something, say so.";
小提示
嘗試不同的系統提示來改變助理的行為。 例如,你可以指示它以海盜、老師或領域專家的身份回應。
實作多回合交談
聊天助理需要在多個交換中維持上下文。 你可以透過保留所有訊息(系統、使用者和助理)的向量,並在每個請求中傳送完整清單來達成這個目標。 模型利用這些歷史來產生具上下文相關的回應。
新增一個對話循環:
- 讀取主機的使用者輸入。
- 將使用者訊息附加到歷史紀錄中。
- 將完整的歷史資料傳送給模型。
- 將助手的回應附加到下一回合的上下文中。
loop {
print!("You: ");
io::stdout().flush()?;
let mut input = String::new();
stdin.lock().read_line(&mut input)?;
let input = input.trim();
if input.eq_ignore_ascii_case("quit") || input.eq_ignore_ascii_case("exit") {
break;
}
if input.is_empty() {
continue;
}
let mut items = Vec::new();
if session.turn_count() == 0 {
items.push(Item::system_message(vec![Item::text(system_prompt)]));
}
items.push(Item::user_message(vec![Item::text(input)]));
print!("Assistant: ");
io::stdout().flush()?;
let mut stream = session.process_streaming_request(Request::from_items(items));
while let Some(item) = stream.next().await {
if let Item::Text {
text,
kind: TextKind::Default,
} = item?
{
print!("{text}");
io::stdout().flush()?;
}
}
println!("\n");
}
每次呼叫 complete_chat 都會收到完整的訊息歷史。 這就是模型「記住」前幾回合的方式——它不會在呼叫之間儲存狀態。
新增串流回應
串流會在每個權杖產生時印出,使助理更能迅速給予回覆。 將complete_chat呼叫替換為complete_streaming_chat以流式傳輸每個回應標記。
更新對話循環以使用串流:
print!("Assistant: ");
io::stdout().flush()?;
let mut stream = session.process_streaming_request(Request::from_items(items));
while let Some(item) = stream.next().await {
if let Item::Text {
text,
kind: TextKind::Default,
} = item?
{
print!("{text}");
io::stdout().flush()?;
}
}
println!("\n");
串流版本會累積完整回應,這樣直播結束後就能加入對話紀錄。
完整程式碼
將 src/main.rs 的內容替換為以下完整程式碼:
use foundry_local_sdk::{
ChatSession, FoundryLocalConfig, FoundryLocalManager, Item, Request, RequestOptions,
SearchOptions, TextKind,
};
use std::io::{self, BufRead, Write};
use tokio_stream::StreamExt;
#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
// Initialize the Foundry Local SDK
let manager = FoundryLocalManager::create(FoundryLocalConfig::new("foundry_local_samples"))?;
manager
.download_and_register_eps_with_progress(None, {
let mut current_ep = String::new();
move |ep_name: &str, percent: f64| {
if ep_name != current_ep {
if !current_ep.is_empty() {
println!();
}
current_ep = ep_name.to_string();
}
print!("\r {ep_name:<30} {percent:5.1}%");
io::stdout().flush().ok();
}
})
.await?;
println!();
let model = manager.catalog().get_model("qwen2.5-0.5b").await?;
if !model.is_cached().await? {
println!("Downloading model...");
model
.download(Some(|progress: f64| {
print!("\r {progress:.1}%");
io::stdout().flush().ok();
}))
.await?;
println!();
}
model.load().await?;
let session = ChatSession::new(&model).await?;
session
.set_options(RequestOptions {
search: SearchOptions {
temperature: Some(0.7),
max_output_tokens: Some(512),
..Default::default()
},
..Default::default()
})
.await?;
println!("Model loaded and ready.");
let system_prompt = "You are a helpful, friendly assistant. Keep your responses concise and \
conversational. If you don't know something, say so.";
println!("\nChat assistant ready! Type 'quit' to exit.\n");
let stdin = io::stdin();
loop {
print!("You: ");
io::stdout().flush()?;
let mut input = String::new();
stdin.lock().read_line(&mut input)?;
let input = input.trim();
if input.eq_ignore_ascii_case("quit") || input.eq_ignore_ascii_case("exit") {
break;
}
if input.is_empty() {
continue;
}
let mut items = Vec::new();
if session.turn_count() == 0 {
items.push(Item::system_message(vec![Item::text(system_prompt)]));
}
items.push(Item::user_message(vec![Item::text(input)]));
print!("Assistant: ");
io::stdout().flush()?;
let mut stream = session.process_streaming_request(Request::from_items(items));
while let Some(item) = stream.next().await {
if let Item::Text {
text,
kind: TextKind::Default,
} = item?
{
print!("{text}");
io::stdout().flush()?;
}
}
println!("\n");
}
drop(session);
model.unload().await?;
println!("Model unloaded. Goodbye!");
Ok(())
}
運行聊天助理:
cargo run
你會看到類似的輸出:
Downloading model: 100.00%
Model loaded and ready.
Chat assistant ready! Type 'quit' to exit.
You: What is photosynthesis?
Assistant: Photosynthesis is the process plants use to convert sunlight, water, and carbon
dioxide into glucose and oxygen. It mainly happens in the leaves, inside structures
called chloroplasts.
You: Why is it important for other living things?
Assistant: It's essential because photosynthesis produces the oxygen that most living things
breathe. It also forms the base of the food chain — animals eat plants or eat other
animals that depend on plants for energy.
You: quit
Model unloaded. Goodbye!
注意助理如何記住之前回合的上下文——當你問「為什麼對其他生物重要?」時,它知道你仍在談論光合作用。
樣本庫
複製 Foundry samples GitHub 存放庫,然後前往 C++ 範例:
git clone https://github.com/microsoft-foundry/foundry-samples.git
cd foundry-samples/samples/cpp/foundry-local/tutorial-chat-assistant
安裝套件
C++ 範例需要:
配置並建立樣本:
cmake -S . -B build
cmake --build build --config Release
在設定過程中,範例會將 2.2.1 版的 Microsoft.AI.Foundry.Local.Runtime NuGet 套件還原到 build/_packages。 該套件包含 Windows、Linux 和 macOS 的 C++ 標頭及原生函式庫。 你不需要另外的 WinML 套件。
CMake 設定會選擇適用於作業系統與架構的執行時資產,並將所需的原生函式庫複製到可執行檔旁邊。 在 macOS 和 Linux 上,從 build 執行可執行檔。 在 Windows 上,執行來自 build 的 .exe 檔案。
瀏覽目錄並選擇型號
初始化 SDK,從型錄中選擇一個模型,然後下載並載入該模型:
Configuration config("foundry_local_cpp_tutorial_chat");
Manager manager(std::move(config));
auto model = SelectChatModel(manager.GetCatalog());
if (!model) {
throw std::runtime_error("No supported CPU chat model was found.");
}
if (!model->IsCached()) {
model->Download([](float progress) {
std::cout << "\rDownloading model: " << static_cast<int>(progress) << "%" << std::flush;
return 0;
});
std::cout << "\n";
}
model->Load();
定義系統提示
新增系統提示來塑造助理的回應方式:
Request system_prompt{
SystemMessage(
"You are a concise, friendly local assistant. State when you are unsure."),
};
session.ProcessRequest(system_prompt);
實作多回合交談
保持對話紀錄,並隨每個請求發送:
std::cout << "Enter a message, or type 'exit'.\n";
for (std::string input; std::cout << "You: " && std::getline(std::cin, input);) {
if (input == "exit" || input == "quit") {
break;
}
if (input.empty()) {
continue;
}
EnableStreaming(session);
Request request{UserMessage(input)};
std::cout << "Assistant: ";
session.ProcessRequest(request);
std::cout << "\n";
}
新增串流回應
以串流方式輸出模型產生的回應文字:
void EnableStreaming(ChatSession& session) {
session.SetStreamingCallback([](Item item) {
if (item.GetType() == FOUNDRY_LOCAL_ITEM_TEXT &&
item.GetText().type == FOUNDRY_LOCAL_TEXT_ITEM_TYPE_DEFAULT) {
std::cout << item.GetText().text << std::flush;
}
return 0;
});
}
完整程式碼
#include <foundry_local/foundry_local_cpp.h>
#include <iostream>
#include <memory>
#include <string>
using namespace foundry_local;
std::unique_ptr<IModel> SelectChatModel(ICatalog& catalog) {
for (const std::string alias : {"qwen2.5-0.5b", "qwen3.5-0.8b"}) {
auto model = catalog.GetModel(alias);
if (!model) {
continue;
}
for (const auto& variant : model->GetVariants()) {
const auto info = variant->GetInfo();
if (info.DeviceType() == FOUNDRY_LOCAL_DEVICE_CPU &&
(info.Task() == "chat-completion" || info.Task() == "coding")) {
model->SelectVariant(*variant);
return model;
}
}
}
return nullptr;
}
void EnableStreaming(ChatSession& session);
int main() {
try {
Configuration config("foundry_local_cpp_tutorial_chat");
Manager manager(std::move(config));
auto model = SelectChatModel(manager.GetCatalog());
if (!model) {
throw std::runtime_error("No supported CPU chat model was found.");
}
if (!model->IsCached()) {
model->Download([](float progress) {
std::cout << "\rDownloading model: " << static_cast<int>(progress) << "%" << std::flush;
return 0;
});
std::cout << "\n";
}
model->Load();
{
ChatSession session(*model);
session.SetOptions({
.search = {.temperature = 0.2F, .max_output_tokens = 512, .seed = 42},
});
Request system_prompt{
SystemMessage(
"You are a concise, friendly local assistant. State when you are unsure."),
};
session.ProcessRequest(system_prompt);
std::cout << "Enter a message, or type 'exit'.\n";
for (std::string input; std::cout << "You: " && std::getline(std::cin, input);) {
if (input == "exit" || input == "quit") {
break;
}
if (input.empty()) {
continue;
}
EnableStreaming(session);
Request request{UserMessage(input)};
std::cout << "Assistant: ";
session.ProcessRequest(request);
std::cout << "\n";
}
}
model->Unload();
manager.Shutdown();
} catch (const Error& error) {
std::cerr << "Foundry Local error [" << error.Code() << "]: " << error.what() << "\n";
return 1;
} catch (const std::exception& error) {
std::cerr << "Error: " << error.what() << "\n";
return 1;
}
return 0;
}
void EnableStreaming(ChatSession& session) {
session.SetStreamingCallback([](Item item) {
if (item.GetType() == FOUNDRY_LOCAL_ITEM_TEXT &&
item.GetText().type == FOUNDRY_LOCAL_TEXT_ITEM_TYPE_DEFAULT) {
std::cout << item.GetText().text << std::flush;
}
return 0;
});
}
運行聊天助理:
./build/tutorial-chat-assistant
清理資源
卸載模型後,模型權重仍保留在本地快取中。 這表示下次執行應用程式時,下載步驟會被跳過,模型載入速度會更快。 除非你想搶回磁碟空間,否則不需要額外清理。