教學:打造語音轉文字筆記器

打造一款應用程式,將語音音訊轉成有組織的筆記——完全在你的裝置上完成。 該應用程式先用語音轉文字模型轉錄音訊檔,接著用聊天模式將轉錄摘要並整理成乾淨的筆記。

在本教學課程中,您將瞭解如何:

  • 建立專案並安裝 Foundry 本地 SDK
  • 載入語音轉文字模型並轉錄音訊檔案
  • 載入聊天模型並摘要文字稿
  • 將轉錄與摘要結合成完整的應用程式
  • 清理資源

先決條件

  • 一台至少有 8GB 記憶體的 Windows、macOS 或 Linux 電腦。
  • 一個 .wav 音訊檔案來轉錄(教學用的是範例檔)。

安裝套件

樣本庫

你可以在 Foundry 範例 GitHub 倉庫找到這篇文章的完整範例程式碼。 要複製儲存庫並導向範例,請使用:

git clone https://github.com/microsoft-foundry/foundry-samples.git
cd foundry-samples/samples/csharp/foundry-local/tutorial-voice-to-text
dotnet add package Microsoft.AI.Foundry.Local
dotnet add package OpenAI

GitHub 倉庫裡的 C# 範例都是預先設定好的專案。 如果你是從零開始建立,建議閱讀 Foundry Local SDK 參考資料 ,了解更多如何用 Foundry Local 設定 C# 專案的細節。

轉錄音檔

在此步驟中,你會載入語音轉文字模型並轉錄音訊檔案。 Foundry Local SDK 利用 whisper 型號別名來選擇最適合你硬體的 Whisper 變體。

  • 打開 Program.cs 並替換為以下程式碼,以初始化 SDK、載入語音模型,並轉錄音訊檔案:

    // Load the speech-to-text model
    var speechModel = await catalog.GetModelAsync("whisper-tiny")
        ?? throw new Exception("Speech model not found");
    
    await speechModel.DownloadAsync(progress =>
    {
        Console.Write($"\rDownloading speech model: {progress:F2}%");
        if (progress >= 100f) Console.WriteLine();
    });
    
    await speechModel.LoadAsync();
    Console.WriteLine("Speech model loaded.");
    
    // Transcribe the audio file with the native v2 audio session.
    var transcriptionText = new StringBuilder();
    using var audioSession = new AudioSession(speechModel);
    audioSession.SetStreaming(true);
    using var audioRequest = new Request();
    audioRequest.AddItem(new AudioItem("meeting-notes.wav"));
    
    Console.WriteLine("\nTranscription:");
    await using var audioResponse = audioSession.ProcessStreamingRequestAsync(audioRequest, ct);
    await foreach (var item in audioResponse)
    {
        using (item)
        {
            if (item is SpeechSegmentItem segment)
            {
                Console.Write(segment.Text);
                transcriptionText.Append(segment.Text);
            }
        }
    }
    using var finalTranscription = await audioResponse.FinalResponse;
    Console.WriteLine();
    
    // Unload the speech model to free memory
    finalTranscription.Dispose();
    await audioResponse.DisposeAsync();
    audioRequest.Dispose();
    audioSession.Dispose();
    await speechModel.UnloadAsync();
    

    GetAudioClientAsync 方法回傳一個用於音訊操作的客戶端。 當轉錄片段可用時,該 TranscribeAudioStreamingAsync 方法會對其進行串流。 你累積文本,以便在下一步將其交給聊天模型。

備註

請以您的音訊檔案路徑取代 "meeting-notes.wav"。 支援的格式包括 WAV、MP3 和 FLAC。

摘要轉錄

現在用聊天模式將原始轉錄整理成結構化的筆記。 載入 qwen2.5-0.5b 模型,把轉錄作為上下文,透過系統提示指示模型產生乾淨且摘要的筆記。

在轉錄步驟後加入以下程式碼:

// Load the chat model for summarization
var chatModel = await catalog.GetModelAsync("qwen2.5-0.5b")
    ?? throw new Exception("Chat model not found");

await chatModel.DownloadAsync(progress =>
{
    Console.Write($"\rDownloading chat model: {progress:F2}%");
    if (progress >= 100f) Console.WriteLine();
});

await chatModel.LoadAsync();
Console.WriteLine("Chat model loaded.");

// Summarize the transcription into organized notes.
using var chatSession = new ChatSession(chatModel);
using var chatRequest = new Request();
chatRequest.AddItem(MessageItem.System(
    "You are a note-taking assistant. Summarize the following transcription " +
    "into organized, concise notes with bullet points."));
chatRequest.AddItem(MessageItem.User(transcriptionText.ToString()));
using var chatResponse = await chatSession.ProcessRequestAsync(chatRequest, ct);
var summary = string.Concat(
    chatResponse.OfType<MessageItem>()
        .SelectMany(message => message.Parts)
        .OfType<TextItem>()
        .Where(text => text.Type == TextItemType.Default)
        .Select(text => text.Text));
Console.WriteLine($"\nSummary:\n{summary}");

// Clean up
chatResponse.Dispose();
chatRequest.Dispose();
chatSession.Dispose();
await chatModel.UnloadAsync();
Console.WriteLine("\nDone. Models unloaded.");

系統提示決定模型的輸出格式。 透過指示它產生「有條理、簡潔且有條列點的筆記」,你得到的是結構化的內容,而非直接的意譯。

合併成完整的應用程式

將Program.cs的內容替換為以下的完整程式碼,該程式碼能轉錄音訊檔案並總結轉錄內容:

using Microsoft.AI.Foundry.Local;
using Microsoft.Extensions.Logging;
using System.Text;

CancellationToken ct = CancellationToken.None;

var config = new Configuration
{
    AppName = "foundry_local_samples",
    LogLevel = Microsoft.AI.Foundry.Local.LogLevel.Information
};

using var loggerFactory = LoggerFactory.Create(builder =>
{
    builder.SetMinimumLevel(
        Microsoft.Extensions.Logging.LogLevel.Information
    );
});
var logger = loggerFactory.CreateLogger<Program>();

// Initialize the singleton instance
await FoundryLocalManager.CreateAsync(config, logger);
var mgr = FoundryLocalManager.Instance;

// Download and register all execution providers.
var currentEp = "";
await mgr.DownloadAndRegisterEpsAsync((epName, percent) =>
{
    if (epName != currentEp)
    {
        if (currentEp != "") Console.WriteLine();
        currentEp = epName;
    }
    Console.Write($"\r  {epName.PadRight(30)}  {percent,6:F1}%");
});
if (currentEp != "") Console.WriteLine();

var catalog = await mgr.GetCatalogAsync();

// Load the speech-to-text model
var speechModel = await catalog.GetModelAsync("whisper-tiny")
    ?? throw new Exception("Speech model not found");

await speechModel.DownloadAsync(progress =>
{
    Console.Write($"\rDownloading speech model: {progress:F2}%");
    if (progress >= 100f) Console.WriteLine();
});

await speechModel.LoadAsync();
Console.WriteLine("Speech model loaded.");

// Transcribe the audio file with the native v2 audio session.
var transcriptionText = new StringBuilder();
using var audioSession = new AudioSession(speechModel);
audioSession.SetStreaming(true);
using var audioRequest = new Request();
audioRequest.AddItem(new AudioItem("meeting-notes.wav"));

Console.WriteLine("\nTranscription:");
await using var audioResponse = audioSession.ProcessStreamingRequestAsync(audioRequest, ct);
await foreach (var item in audioResponse)
{
    using (item)
    {
        if (item is SpeechSegmentItem segment)
        {
            Console.Write(segment.Text);
            transcriptionText.Append(segment.Text);
        }
    }
}
using var finalTranscription = await audioResponse.FinalResponse;
Console.WriteLine();

// Unload the speech model to free memory
finalTranscription.Dispose();
await audioResponse.DisposeAsync();
audioRequest.Dispose();
audioSession.Dispose();
await speechModel.UnloadAsync();

// Load the chat model for summarization
var chatModel = await catalog.GetModelAsync("qwen2.5-0.5b")
    ?? throw new Exception("Chat model not found");

await chatModel.DownloadAsync(progress =>
{
    Console.Write($"\rDownloading chat model: {progress:F2}%");
    if (progress >= 100f) Console.WriteLine();
});

await chatModel.LoadAsync();
Console.WriteLine("Chat model loaded.");

// Summarize the transcription into organized notes.
using var chatSession = new ChatSession(chatModel);
using var chatRequest = new Request();
chatRequest.AddItem(MessageItem.System(
    "You are a note-taking assistant. Summarize the following transcription " +
    "into organized, concise notes with bullet points."));
chatRequest.AddItem(MessageItem.User(transcriptionText.ToString()));
using var chatResponse = await chatSession.ProcessRequestAsync(chatRequest, ct);
var summary = string.Concat(
    chatResponse.OfType<MessageItem>()
        .SelectMany(message => message.Parts)
        .OfType<TextItem>()
        .Where(text => text.Type == TextItemType.Default)
        .Select(text => text.Text));
Console.WriteLine($"\nSummary:\n{summary}");

// Clean up
chatResponse.Dispose();
chatRequest.Dispose();
chatSession.Dispose();
await chatModel.UnloadAsync();
Console.WriteLine("\nDone. Models unloaded.");

備註

請以您的音訊檔案路徑取代 "meeting-notes.wav"。 支援的格式包括 WAV、MP3 和 FLAC。

執行筆記應用程式:

dotnet run

你會看到類似的輸出:

Downloading speech model: 100.00%
Speech model loaded.

Transcription:
OK so let's get started with the weekly sync. First, the backend
API is nearly done. Sarah finished the authentication endpoints
yesterday. We still need to add rate limiting before we go to
staging. On the frontend, the dashboard redesign is about seventy
percent complete. Jake, can you walk us through the new layout?
Great. The charts look good. I think we should add a filter for
date range though. For testing, we have about eighty percent code
coverage on the API. We need to write integration tests for the
new auth flow before Friday. Let's plan to do a full regression
test next Tuesday before the release. Any blockers? OK, sounds
like we are in good shape. Let's wrap up.

Downloading chat model: 100.00%
Chat model loaded.

Summary:
- **Backend API**: Authentication endpoints complete. Rate limiting
  still needed before staging deployment.
- **Frontend**: Dashboard redesign 70% complete. New chart layout
  reviewed. Action item: add a date range filter.
- **Testing**: API code coverage at 80%. Integration tests for the
  auth flow due Friday. Full regression test scheduled for next
  Tuesday before release.
- **Status**: No blockers reported. Team is on track.

Done. Models unloaded.

應用程式先將音訊內容轉錄並串流輸出,然後將累積的文字傳送給聊天模型,該模型從中提取關鍵點並將其組織成結構化的筆記。

安裝套件

樣本庫

本文的完整範例程式碼可在 Foundry-samples GitHub 倉庫取得。 要複製儲存庫並導向範例,請使用:

git clone https://github.com/microsoft-foundry/foundry-samples.git
cd foundry-samples/samples/javascript/foundry-local/tutorial-voice-to-text
npm install foundry-local-sdk openai

轉錄音檔

在此步驟中,你會載入語音轉文字模型並轉錄音訊檔案。 Foundry Local SDK 利用 whisper 型號別名來選擇最適合你硬體的 Whisper 變體。

  1. 建立一個名為 app.js. 的檔案。

  2. 新增以下程式碼以初始化 SDK、載入語音模型,並轉錄音訊檔案:

    // Load the speech-to-text model
    const speechModel = await manager.catalog.getModel('whisper-tiny');
    await speechModel.download((progress) => {
        process.stdout.write(
            `\rDownloading speech model: ${progress.toFixed(2)}%`
        );
    });
    console.log('\nSpeech model downloaded.');
    
    await speechModel.load();
    console.log('Speech model loaded.');
    
    // Transcribe the audio file
    const audioSession = new AudioSession(speechModel);
    const audioFile = path.join(__dirname, 'meeting-notes.wav');
    const transcriptionResponse = await audioSession.processRequest(
        new Request().addItem(Item.audioFromUri(audioFile))
    );
    const transcription = transcriptionResponse.output.find(
        (item) => item.type === 'speechResult'
    );
    if (!transcription) {
        throw new Error('The speech model returned no transcription.');
    }
    console.log(`\nTranscription:\n${transcription.text}`);
    
    // Unload the speech model to free memory
    audioSession.dispose();
    await speechModel.unload();
    

    createAudioClient 方法回傳一個用於音訊操作的客戶端。 此 transcribe 方法接受檔案路徑,並回傳一個包含 text 轉錄內容屬性的物件。

備註

請以您的音訊檔案路徑取代 './meeting-notes.wav'。 支援的格式包括 WAV、MP3 和 FLAC。

摘要轉錄

現在用聊天模式將原始轉錄整理成結構化的筆記。 載入 qwen2.5-0.5b 模型,把轉錄作為上下文,透過系統提示指示模型產生乾淨且摘要的筆記。

在轉錄步驟後加入以下程式碼:

// Load the chat model for summarization
const chatModel = await manager.catalog.getModel('qwen2.5-0.5b');
await chatModel.download((progress) => {
    process.stdout.write(
        `\rDownloading chat model: ${progress.toFixed(2)}%`
    );
});
console.log('\nChat model downloaded.');

await chatModel.load();
console.log('Chat model loaded.');

// Summarize the transcription into organized notes
const chatSession = new ChatSession(chatModel);
const request = new Request()
    .addItem(Item.systemMessage(
        'You are a note-taking assistant. Summarize ' +
        'the following transcription into organized, ' +
        'concise notes with bullet points.'
    ))
    .addItem(Item.userMessage(transcription.text));

const response = await chatSession.processRequest(request);
const summary = response.output
    .map((item) => item.type === 'message'
        ? item.content ?? item.parts?.map((part) => part.text ?? '').join('') ?? ''
        : item.type === 'text' ? item.text : '')
    .join('');
console.log(`\nSummary:\n${summary}`);

// Clean up
chatSession.dispose();
await chatModel.unload();
console.log('\nDone. Models unloaded.');

系統提示決定模型的輸出格式。 透過指示它產生「有條理、簡潔且有條列點的筆記」,你得到的是結構化的內容,而非直接的意譯。

合併成完整的應用程式

建立一個名為 app.js S 的檔案,並加入以下完整程式碼,將音訊檔案轉錄並總結:

import {
    AudioSession,
    ChatSession,
    FoundryLocalManager,
    Item,
    Request
} from 'foundry-local-sdk';
import { fileURLToPath } from 'url';
import path from 'path';

const __dirname = path.dirname(fileURLToPath(import.meta.url));

// Initialize the Foundry Local SDK
const manager = FoundryLocalManager.create({
    appName: 'foundry_local_samples',
    logLevel: 'info'
});

// Download and register all execution providers.
let currentEp = '';
await manager.downloadAndRegisterEps((epName, percent) => {
    if (epName !== currentEp) {
        if (currentEp !== '') process.stdout.write('\n');
        currentEp = epName;
    }
    process.stdout.write(`\r  ${epName.padEnd(30)}  ${percent.toFixed(1).padStart(5)}%`);
});
if (currentEp !== '') process.stdout.write('\n');

// Load the speech-to-text model
const speechModel = await manager.catalog.getModel('whisper-tiny');
await speechModel.download((progress) => {
    process.stdout.write(
        `\rDownloading speech model: ${progress.toFixed(2)}%`
    );
});
console.log('\nSpeech model downloaded.');

await speechModel.load();
console.log('Speech model loaded.');

// Transcribe the audio file
const audioSession = new AudioSession(speechModel);
const audioFile = path.join(__dirname, 'meeting-notes.wav');
const transcriptionResponse = await audioSession.processRequest(
    new Request().addItem(Item.audioFromUri(audioFile))
);
const transcription = transcriptionResponse.output.find(
    (item) => item.type === 'speechResult'
);
if (!transcription) {
    throw new Error('The speech model returned no transcription.');
}
console.log(`\nTranscription:\n${transcription.text}`);

// Unload the speech model to free memory
audioSession.dispose();
await speechModel.unload();

// Load the chat model for summarization
const chatModel = await manager.catalog.getModel('qwen2.5-0.5b');
await chatModel.download((progress) => {
    process.stdout.write(
        `\rDownloading chat model: ${progress.toFixed(2)}%`
    );
});
console.log('\nChat model downloaded.');

await chatModel.load();
console.log('Chat model loaded.');

// Summarize the transcription into organized notes
const chatSession = new ChatSession(chatModel);
const request = new Request()
    .addItem(Item.systemMessage(
        'You are a note-taking assistant. Summarize ' +
        'the following transcription into organized, ' +
        'concise notes with bullet points.'
    ))
    .addItem(Item.userMessage(transcription.text));

const response = await chatSession.processRequest(request);
const summary = response.output
    .map((item) => item.type === 'message'
        ? item.content ?? item.parts?.map((part) => part.text ?? '').join('') ?? ''
        : item.type === 'text' ? item.text : '')
    .join('');
console.log(`\nSummary:\n${summary}`);

// Clean up
chatSession.dispose();
await chatModel.unload();
console.log('\nDone. Models unloaded.');

備註

請以您的音訊檔案路徑取代 './meeting-notes.wav'。 支援的格式包括 WAV、MP3 和 FLAC。

執行筆記應用程式:

node app.js

你會看到類似的輸出:

Downloading speech model: 100.00%
Speech model downloaded.
Speech model loaded.

Transcription:
OK so let's get started with the weekly sync. First, the backend
API is nearly done. Sarah finished the authentication endpoints
yesterday. We still need to add rate limiting before we go to
staging. On the frontend, the dashboard redesign is about seventy
percent complete. Jake, can you walk us through the new layout?
Great. The charts look good. I think we should add a filter for
date range though. For testing, we have about eighty percent code
coverage on the API. We need to write integration tests for the
new auth flow before Friday. Let's plan to do a full regression
test next Tuesday before the release. Any blockers? OK, sounds
like we are in good shape. Let's wrap up.

Downloading chat model: 100.00%
Chat model downloaded.
Chat model loaded.

Summary:
- **Backend API**: Authentication endpoints complete. Rate limiting
  still needed before staging deployment.
- **Frontend**: Dashboard redesign 70% complete. New chart layout
  reviewed. Action item: add a date range filter.
- **Testing**: API code coverage at 80%. Integration tests for the
  auth flow due Friday. Full regression test scheduled for next
  Tuesday before release.
- **Status**: No blockers reported. Team is on track.

Done. Models unloaded.

應用程式會先將音訊內容轉錄,然後將文字傳給聊天模型,該模型能擷取關鍵點並將其組織成結構化的筆記。

安裝套件

樣本庫

本文的完整範例程式碼可在 Foundry-samples GitHub 倉庫取得。 要複製儲存庫並導向範例,請使用:

git clone https://github.com/microsoft-foundry/foundry-samples.git
cd foundry-samples/samples/python/foundry-local/tutorial-voice-to-text
pip install foundry-local-sdk openai

轉錄音檔

在此步驟中,你會載入語音轉文字模型並轉錄音訊檔案。 Foundry Local SDK 利用 whisper 型號別名來選擇最適合你硬體的 Whisper 變體。

  1. 建立一個名為 app.py. 的檔案。

  2. 新增以下程式碼以初始化 SDK、載入語音模型,並轉錄音訊檔案:

    # Load the speech-to-text model.
    speech_model = manager.catalog.get_model("whisper-tiny")
    if speech_model is None:
        raise RuntimeError("Model 'whisper-tiny' was not found in the catalog.")
    download_if_needed(speech_model, "speech")
    speech_model.load()
    print("Speech model loaded.")
    
    try:
        # Transcribe the audio file with a native audio session.
        with AudioSession(speech_model) as session:
            with Request().add_item(AudioItem.from_uri(audio_path)) as request:
                with session.process_request(request) as response:
                    transcription = get_text(response)
        print(f"\nTranscription:\n{transcription}")
    finally:
        # Unload the speech model to free memory.
        if speech_model.is_loaded:
            speech_model.unload()
    

    get_audio_client 方法回傳一個用於音訊操作的客戶端。 此 transcribe 方法接受檔案路徑,並回傳一個包含 text 轉錄內容屬性的物件。

備註

請以您的音訊檔案路徑取代 "meeting-notes.wav"。 支援的格式包括 WAV、MP3 和 FLAC。

摘要轉錄

現在用聊天模式將原始轉錄整理成結構化的筆記。 載入 qwen2.5-0.5b 模型,把轉錄作為上下文,透過系統提示指示模型產生乾淨且摘要的筆記。

在轉錄步驟後加入以下程式碼:

# Load the chat model for summarization.
chat_model = manager.catalog.get_model("qwen2.5-0.5b")
if chat_model is None:
    manager.close()
    raise RuntimeError("Model 'qwen2.5-0.5b' was not found in the catalog.")
download_if_needed(chat_model, "chat")
chat_model.load()
print("Chat model loaded.")

try:
    # Summarize the transcription into organized notes.
    with ChatSession(chat_model) as session:
        session.set_options(
            RequestOptions(
                search=SearchOptions(temperature=0.2, max_output_tokens=512)
            )
        )
        with (
            Request()
            .add_item(
                MessageItem.system(
                    "You are a note-taking assistant. Summarize the following "
                    "transcription into organized, concise notes with bullet points."
                )
            )
            .add_item(MessageItem.user(transcription)) as request
        ):
            with session.process_request(request) as response:
                summary = get_text(response)
    print(f"\nSummary:\n{summary}")
finally:
    # Clean up.
    if chat_model.is_loaded:
        chat_model.unload()
    manager.close()
    print("\nDone. Models unloaded.")

系統提示決定模型的輸出格式。 透過指示它產生「有條理、簡潔且有條列點的筆記」,你得到的是結構化的內容,而非直接的意譯。

合併成完整的應用程式

建立一個名為 app.py S 的檔案,並加入以下完整程式碼,將音訊檔案轉錄並總結:

import sys

from foundry_local_sdk import (
    AudioItem,
    AudioSession,
    ChatSession,
    Configuration,
    FoundryLocalManager,
    MessageItem,
    Request,
    RequestOptions,
    SearchOptions,
    SpeechResultItem,
    TextItem,
)



def download_if_needed(model, label):
    if model.is_cached:
        print(f"Using cached {label} model {model.id}.")
        return
    model.download(
        lambda progress: print(
            f"\rDownloading {label} model: {progress:.2f}%",
            end="",
            flush=True,
        )
    )
    print()


def get_text(response):
    parts = []
    for item in response:
        if isinstance(item, SpeechResultItem):
            parts.append(item.text)
        elif isinstance(item, TextItem):
            parts.append(item.text)
        elif isinstance(item, MessageItem):
            parts.extend(part.text for part in item.parts if isinstance(part, TextItem))
    return "".join(parts)


def main():
    # Initialize the Foundry Local SDK
    config = Configuration(app_name="foundry_local_samples")
    FoundryLocalManager.initialize(config)
    manager = FoundryLocalManager.instance

    audio_path = sys.argv[1] if len(sys.argv) > 1 else "meeting-notes.wav"

    # Load the speech-to-text model.
    speech_model = manager.catalog.get_model("whisper-tiny")
    if speech_model is None:
        raise RuntimeError("Model 'whisper-tiny' was not found in the catalog.")
    download_if_needed(speech_model, "speech")
    speech_model.load()
    print("Speech model loaded.")

    try:
        # Transcribe the audio file with a native audio session.
        with AudioSession(speech_model) as session:
            with Request().add_item(AudioItem.from_uri(audio_path)) as request:
                with session.process_request(request) as response:
                    transcription = get_text(response)
        print(f"\nTranscription:\n{transcription}")
    finally:
        # Unload the speech model to free memory.
        if speech_model.is_loaded:
            speech_model.unload()

    # Load the chat model for summarization.
    chat_model = manager.catalog.get_model("qwen2.5-0.5b")
    if chat_model is None:
        manager.close()
        raise RuntimeError("Model 'qwen2.5-0.5b' was not found in the catalog.")
    download_if_needed(chat_model, "chat")
    chat_model.load()
    print("Chat model loaded.")

    try:
        # Summarize the transcription into organized notes.
        with ChatSession(chat_model) as session:
            session.set_options(
                RequestOptions(
                    search=SearchOptions(temperature=0.2, max_output_tokens=512)
                )
            )
            with (
                Request()
                .add_item(
                    MessageItem.system(
                        "You are a note-taking assistant. Summarize the following "
                        "transcription into organized, concise notes with bullet points."
                    )
                )
                .add_item(MessageItem.user(transcription)) as request
            ):
                with session.process_request(request) as response:
                    summary = get_text(response)
        print(f"\nSummary:\n{summary}")
    finally:
        # Clean up.
        if chat_model.is_loaded:
            chat_model.unload()
        manager.close()
        print("\nDone. Models unloaded.")


if __name__ == "__main__":
    main()

備註

請以您的音訊檔案路徑取代 "meeting-notes.wav"。 支援的格式包括 WAV、MP3 和 FLAC。

執行筆記應用程式:

python app.py

你會看到類似的輸出:

Downloading speech model: 100.00%
Speech model loaded.

Transcription:
OK so let's get started with the weekly sync. First, the backend
API is nearly done. Sarah finished the authentication endpoints
yesterday. We still need to add rate limiting before we go to
staging. On the frontend, the dashboard redesign is about seventy
percent complete. Jake, can you walk us through the new layout?
Great. The charts look good. I think we should add a filter for
date range though. For testing, we have about eighty percent code
coverage on the API. We need to write integration tests for the
new auth flow before Friday. Let's plan to do a full regression
test next Tuesday before the release. Any blockers? OK, sounds
like we are in good shape. Let's wrap up.

Downloading chat model: 100.00%
Chat model loaded.

Summary:
- **Backend API**: Authentication endpoints complete. Rate limiting
  still needed before staging deployment.
- **Frontend**: Dashboard redesign 70% complete. New chart layout
  reviewed. Action item: add a date range filter.
- **Testing**: API code coverage at 80%. Integration tests for the
  auth flow due Friday. Full regression test scheduled for next
  Tuesday before release.
- **Status**: No blockers reported. Team is on track.

Done. Models unloaded.

應用程式會先將音訊內容轉錄,然後將文字傳給聊天模型,該模型能擷取關鍵點並將其組織成結構化的筆記。

安裝套件

樣本庫

本文的完整範例程式碼可在 Foundry-samples GitHub 倉庫取得。 要複製儲存庫並導向範例,請使用:

git clone https://github.com/microsoft-foundry/foundry-samples.git
cd foundry-samples/samples/rust/foundry-local/tutorial-voice-to-text
cargo add foundry-local-sdk
cargo add tokio --features full
cargo add tokio-stream anyhow

轉錄音檔

在此步驟中,你會載入語音轉文字模型並轉錄音訊檔案。 Foundry Local SDK 利用 whisper 型號別名來選擇最適合你硬體的 Whisper 變體。

  • 打開 src/main.rs 並替換為以下程式碼,以初始化 SDK、載入語音模型,並轉錄音訊檔案:

    let speech_model = manager.catalog().get_model("whisper-tiny").await?;
    
    if !speech_model.is_cached().await? {
        println!("Downloading speech model...");
        speech_model
            .download(Some(|progress: f64| {
                print!("\r  {progress:.1}%");
                io::stdout().flush().ok();
            }))
            .await?;
        println!();
    }
    
    speech_model.load().await?;
    println!("Speech model loaded.");
    
    let audio_session = AudioSession::new(&speech_model).await?;
    let audio_path = std::fs::canonicalize("meeting-notes.wav")?;
    let audio_uri = Url::from_file_path(&audio_path)
        .map_err(|_| format!("Could not convert {} to a file URI.", audio_path.display()))?;
    let audio = Item::audio_uri(audio_uri.as_str(), Some("wav"), 0, 0);
    let transcription = audio_session.transcribe(audio).await?;
    println!("\nTranscription:\n{transcription}");
    
    drop(audio_session);
    speech_model.unload().await?;
    

    create_audio_client 方法回傳一個用於音訊操作的客戶端。 此 transcribe 方法接受檔案路徑,並回傳一個包含 text 轉錄內容欄位的物件。

備註

請以您的音訊檔案路徑取代 "meeting-notes.wav"。 支援的格式包括 WAV、MP3 和 FLAC。

摘要轉錄

現在用聊天模式將原始轉錄整理成結構化的筆記。 載入 qwen2.5-0.5b 模型,把轉錄作為上下文,透過系統提示指示模型產生乾淨且摘要的筆記。

在轉錄步驟結束後,於函式main中新增以下程式碼:

let chat_model = manager.catalog().get_model("qwen2.5-0.5b").await?;

if !chat_model.is_cached().await? {
    println!("Downloading chat model...");
    chat_model
        .download(Some(|progress: f64| {
            print!("\r  {progress:.1}%");
            io::stdout().flush().ok();
        }))
        .await?;
    println!();
}

chat_model.load().await?;
println!("Chat model loaded.");

let chat_session = ChatSession::new(&chat_model).await?;
chat_session
    .set_options(RequestOptions {
        search: SearchOptions {
            temperature: Some(0.7),
            max_output_tokens: Some(512),
            ..Default::default()
        },
        ..Default::default()
    })
    .await?;

let response = chat_session
    .process_request(Request::from_items(vec![
        Item::system_message(vec![Item::text(
            "You are a note-taking assistant. Summarize the following transcription into \
             organized, concise notes with bullet points.",
        )]),
        Item::user_message(vec![Item::text(transcription)]),
    ]))
    .await?;

println!("\nSummary:");
for item in &response.items {
    if let Some(message) = item.as_message() {
        print!("{}", message.text());
    } else if let Some(text) = item.as_text() {
        print!("{text}");
    }
}
println!();

drop(chat_session);
chat_model.unload().await?;
println!("\nDone. Models unloaded.");

系統提示決定模型的輸出格式。 透過指示它產生「有條理、簡潔且有條列點的筆記」,你得到的是結構化的內容,而非直接的意譯。

合併成完整的應用程式

將src/main.rs的內容替換為以下的完整程式碼,該程式碼能轉錄音訊檔案並總結轉錄內容:

use foundry_local_sdk::{
    AudioSession, ChatSession, FoundryLocalConfig, FoundryLocalManager, Item, Request,
    RequestOptions, SearchOptions,
};
use std::io::{self, Write};
use url::Url;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Initialize the Foundry Local SDK
    let manager = FoundryLocalManager::create(FoundryLocalConfig::new("foundry_local_samples"))?;

    manager
        .download_and_register_eps_with_progress(None, {
            let mut current_ep = String::new();
            move |ep_name: &str, percent: f64| {
                if ep_name != current_ep {
                    if !current_ep.is_empty() {
                        println!();
                    }
                    current_ep = ep_name.to_string();
                }
                print!("\r  {ep_name:<30}  {percent:5.1}%");
                io::stdout().flush().ok();
            }
        })
        .await?;
    println!();

    let speech_model = manager.catalog().get_model("whisper-tiny").await?;

    if !speech_model.is_cached().await? {
        println!("Downloading speech model...");
        speech_model
            .download(Some(|progress: f64| {
                print!("\r  {progress:.1}%");
                io::stdout().flush().ok();
            }))
            .await?;
        println!();
    }

    speech_model.load().await?;
    println!("Speech model loaded.");

    let audio_session = AudioSession::new(&speech_model).await?;
    let audio_path = std::fs::canonicalize("meeting-notes.wav")?;
    let audio_uri = Url::from_file_path(&audio_path)
        .map_err(|_| format!("Could not convert {} to a file URI.", audio_path.display()))?;
    let audio = Item::audio_uri(audio_uri.as_str(), Some("wav"), 0, 0);
    let transcription = audio_session.transcribe(audio).await?;
    println!("\nTranscription:\n{transcription}");

    drop(audio_session);
    speech_model.unload().await?;

    let chat_model = manager.catalog().get_model("qwen2.5-0.5b").await?;

    if !chat_model.is_cached().await? {
        println!("Downloading chat model...");
        chat_model
            .download(Some(|progress: f64| {
                print!("\r  {progress:.1}%");
                io::stdout().flush().ok();
            }))
            .await?;
        println!();
    }

    chat_model.load().await?;
    println!("Chat model loaded.");

    let chat_session = ChatSession::new(&chat_model).await?;
    chat_session
        .set_options(RequestOptions {
            search: SearchOptions {
                temperature: Some(0.7),
                max_output_tokens: Some(512),
                ..Default::default()
            },
            ..Default::default()
        })
        .await?;

    let response = chat_session
        .process_request(Request::from_items(vec![
            Item::system_message(vec![Item::text(
                "You are a note-taking assistant. Summarize the following transcription into \
                 organized, concise notes with bullet points.",
            )]),
            Item::user_message(vec![Item::text(transcription)]),
        ]))
        .await?;

    println!("\nSummary:");
    for item in &response.items {
        if let Some(message) = item.as_message() {
            print!("{}", message.text());
        } else if let Some(text) = item.as_text() {
            print!("{text}");
        }
    }
    println!();

    drop(chat_session);
    chat_model.unload().await?;
    println!("\nDone. Models unloaded.");

    Ok(())
}

備註

請以您的音訊檔案路徑取代 "meeting-notes.wav"。 支援的格式包括 WAV、MP3 和 FLAC。

執行筆記應用程式:

cargo run

你會看到類似的輸出:

Downloading speech model: 100.00%
Speech model loaded.

Transcription:
OK so let's get started with the weekly sync. First, the backend
API is nearly done. Sarah finished the authentication endpoints
yesterday. We still need to add rate limiting before we go to
staging. On the frontend, the dashboard redesign is about seventy
percent complete. Jake, can you walk us through the new layout?
Great. The charts look good. I think we should add a filter for
date range though. For testing, we have about eighty percent code
coverage on the API. We need to write integration tests for the
new auth flow before Friday. Let's plan to do a full regression
test next Tuesday before the release. Any blockers? OK, sounds
like we are in good shape. Let's wrap up.

Downloading chat model: 100.00%
Chat model loaded.

Summary:
- **Backend API**: Authentication endpoints complete. Rate limiting
  still needed before staging deployment.
- **Frontend**: Dashboard redesign 70% complete. New chart layout
  reviewed. Action item: add a date range filter.
- **Testing**: API code coverage at 80%. Integration tests for the
  auth flow due Friday. Full regression test scheduled for next
  Tuesday before release.
- **Status**: No blockers reported. Team is on track.

Done. Models unloaded.

應用程式會先將音訊內容轉錄,然後將文字傳給聊天模型,該模型能擷取關鍵點並將其組織成結構化的筆記。

樣本庫

複製 Foundry samples GitHub 存放庫,然後前往 C++ 範例:

git clone https://github.com/microsoft-foundry/foundry-samples.git
cd foundry-samples/samples/cpp/foundry-local/tutorial-voice-to-text

安裝套件

C++ 範例需要:

  • CMake 3.20 或更新版本。
  • 一個 C++20 編譯器。
  • PATH 上的 NuGet CLI。

配置並建置範例:

cmake -S . -B build
cmake --build build --config Release

在組態期間,範本會將 2.2.1 版的 Microsoft.AI.Foundry.Local.RuntimeNuGet 套件還原到 build/_packages。 該套件包含 Windows、Linux 和 macOS 的 C++ 標頭及原生函式庫。 你不需要另外的 WinML 套件。

CMake 設定會選擇作業系統與架構的執行時資產,並將所需的原生函式庫複製到執行檔旁。 在 macOS 和 Linux 上,請從 build 執行該可執行檔。 在 Windows 上,從 build 執行 .exe 檔案。

轉錄音檔

載入語音模型並轉錄音訊檔案:

auto speech_model = manager.GetCatalog().GetModel("whisper-tiny");
if (!speech_model) {
  throw std::runtime_error("Model 'whisper-tiny' was not found in the catalog.");
}
DownloadIfNeeded(*speech_model, "speech");
speech_model->Load();

std::string transcript;
{
  AudioSession session(*speech_model);
  Request request{Item::AudioFromUri(argv[1])};
  transcript = ResponseText(session.ProcessRequest(request));
}
speech_model->Unload();
std::cout << "Transcription:\n" << transcript << "\n";

摘要轉錄

載入聊天模型,將轉錄文字轉為結構化筆記:

auto chat_model = manager.GetCatalog().GetModel("qwen2.5-0.5b");
if (!chat_model) {
  throw std::runtime_error("Model 'qwen2.5-0.5b' was not found in the catalog.");
}
DownloadIfNeeded(*chat_model, "chat");
chat_model->Load();

{
  ChatSession session(*chat_model);
  Request request{
      SystemMessage(
          "Turn the transcription into concise meeting notes with action items."),
      UserMessage(transcript),
  };
  request.SetOptions({
      .search = {.temperature = 0.2F, .max_output_tokens = 512, .seed = 42},
  });
  std::cout << "Summary:\n" << ResponseText(session.ProcessRequest(request)) << "\n";
}
chat_model->Unload();

合併步驟

#include <foundry_local/foundry_local_cpp.h>

#include <iostream>
#include <string>

using namespace foundry_local;

void DownloadIfNeeded(IModel& model, const std::string& label) {
  if (model.IsCached()) {
    std::cout << "Using cached " << label << " model.\n";
    return;
  }
  model.Download([&label](float progress) {
    std::cout << "\rDownloading " << label << " model: " << static_cast<int>(progress) << "%"
              << std::flush;
    return 0;
  });
  std::cout << "\n";
}

std::string ResponseText(const Response& response) {
  std::string text;
  for (const Item& item : response.GetItems()) {
    if (item.GetType() == FOUNDRY_LOCAL_ITEM_SPEECH_RESULT) {
      text.append(item.GetSpeechResult().text);
    } else if (item.GetType() == FOUNDRY_LOCAL_ITEM_TEXT) {
      text.append(item.GetText().text);
    } else if (item.GetType() == FOUNDRY_LOCAL_ITEM_MESSAGE) {
      for (const Item& part : item.GetMessage().parts) {
        if (part.GetType() == FOUNDRY_LOCAL_ITEM_TEXT) {
          text.append(part.GetText().text);
        }
      }
    }
  }
  return text;
}

int main(int argc, char* argv[]) {
  if (argc != 2) {
    std::cerr << "Usage: tutorial-voice-to-text <audio-file>\n";
    return 1;
  }

  try {
    Configuration config("foundry_local_cpp_voice_to_text");
    Manager manager(std::move(config));

    auto speech_model = manager.GetCatalog().GetModel("whisper-tiny");
    if (!speech_model) {
      throw std::runtime_error("Model 'whisper-tiny' was not found in the catalog.");
    }
    DownloadIfNeeded(*speech_model, "speech");
    speech_model->Load();

    std::string transcript;
    {
      AudioSession session(*speech_model);
      Request request{Item::AudioFromUri(argv[1])};
      transcript = ResponseText(session.ProcessRequest(request));
    }
    speech_model->Unload();
    std::cout << "Transcription:\n" << transcript << "\n";

    auto chat_model = manager.GetCatalog().GetModel("qwen2.5-0.5b");
    if (!chat_model) {
      throw std::runtime_error("Model 'qwen2.5-0.5b' was not found in the catalog.");
    }
    DownloadIfNeeded(*chat_model, "chat");
    chat_model->Load();

    {
      ChatSession session(*chat_model);
      Request request{
          SystemMessage(
              "Turn the transcription into concise meeting notes with action items."),
          UserMessage(transcript),
      };
      request.SetOptions({
          .search = {.temperature = 0.2F, .max_output_tokens = 512, .seed = 42},
      });
      std::cout << "Summary:\n" << ResponseText(session.ProcessRequest(request)) << "\n";
    }
    chat_model->Unload();

    manager.Shutdown();
  } catch (const Error& error) {
    std::cerr << "Foundry Local error [" << error.Code() << "]: " << error.what() << "\n";
    return 1;
  } catch (const std::exception& error) {
    std::cerr << "Error: " << error.what() << "\n";
    return 1;
  }

  return 0;
}

使用音訊檔案路徑執行筆記記錄工具:

./build/tutorial-voice-to-text path/to/audio.wav

清理資源

卸載模型後,模型權重仍保留在本地快取中。 這表示下次執行應用程式時,下載步驟會被跳過,模型載入速度會更快。 除非你想搶回磁碟空間,否則不需要額外清理。