Language
Edit

Recognize speech with your own model using Windows ML

Important

The Windows ML Runtime and Speech Recognition APIs are currently experimental and not supported for use in production environments. Apps trying out these APIs should not be published to the Microsoft Store.

Important

The Windows ML Runtime and Speech Recognition APIs are currently available in C++ and Python. C# is supported through a temporary wrapper that you copy into your project; a future update will include the C# APIs natively. See Set up a project.

The Windows ML Speech Recognition API simplifies running automatic speech recognition (ASR) over your own speech model, on top of the Windows ML Runtime API. It handles the transcription loop, streaming output, and cancellation for you. Windows doesn't ship a model for this API; you choose, package, and distribute the model with your app. It's one of the Windows ML Task APIs.

Tip

If you don't need a specific speech model or custom deployment and just want on-device speech-to-text with a model Microsoft manages, use the ready-to-use Speech Recognition API instead.

There are two ways to specify a model for speech recognition:

  • From a checkpoint folder (Windows ML 2.9). Load a published speech model folder, such as Whisper or Nemotron, with LoadModelFromPath and call SetModel with a GPU execution target. Windows ML compiles the model for the GPU with the CGC backend. Pass the model's tokenizer explicitly. See Choose how a Task runs your model.
  • From pipelines you build. For a Whisper model, create the underlying Runtime objects — the encoder and decoder models, the execution target, and the pipelines — and pass them to the session. This path doesn't hide hardware placement; it only removes the need to write your own decode loop.

Supported models

The checkpoint-folder path takes a published speech model folder. The folder needs config.json, the weights in model.safetensors (or model.safetensors.index.json with every shard it names), and the tokenizer files, such as tokenizer.json and tokenizer_config.json. A Whisper folder also needs generation_config.json, unless config.json holds no_timestamps_token_id. A model that Windows ML doesn't support fails with HRESULT_FROM_WIN32(ERROR_NOT_SUPPORTED).

The pipeline path is specific to the Whisper encoder/decoder architecture. You supply ONNX encoder and decoder models (encoder_model.onnx and decoder_model.onnx), which run through ONNX Runtime, plus a tokenizer file.

Sessions from a checkpoint folder accept mono waveforms at the model's sample rate, or at a rate from 8,000 through 48,000 Hz when the model converts its input. The input tensor holds mono floating-point PCM samples plus sample-rate and valid-length metadata.

Note

When Windows ML runs a model with the CGC backend, start with the recommended speech models: Whisper Medium, Whisper tiny (the samples' model), and Nemotron 3.5 ASR Streaming 0.6B. Other speech models aren't validated yet.

Live input

A model that accepts audio as your app records it, such as a streaming model like Nemotron, exposes a live session (IWinMLAutomaticSpeechRecognitionLiveSession). You start live input, write audio chunks as you capture them, and read transcript fragments as the model commits them. Live input uses bounded backpressure: WriteChunk reports whether the Task accepted the chunk. Query the session for the live interface to test support, because a model that recognizes speech in fixed-length windows, such as Whisper, doesn't expose it.

Language, mode, and predictions

A speech recognition configuration can set a language, a mode, and whether to report predictions. Language support depends on the model.

  • Mode. TRANSCRIBE, the default, writes the speech as text in the language spoken. TRANSLATE writes it as English text. Sessions created from a checkpoint folder transcribe only in Windows ML 2.9, so TRANSLATE fails CreateSession with HRESULT_FROM_WIN32(ERROR_NOT_SUPPORTED). The encoder-decoder composition translates when its tokenizer maps <|translate|> to identifier 50358.
  • Predictions. Opt in to provisional text that follows the recognized text and that later recognition might revise. It's off by default. A model that recognizes speech in fixed-length windows, such as Whisper, and the encoder-decoder composition report no predictions. Without predictions, each update delivers the text of a segment as the model commits it, and Windows ML never revises it. Every segment after the first starts with one space, so the fragments of an operation concatenate into its transcript.

Note

The Whisper-specific speech recognition names in Windows ML 2.7, such as the Whisper configuration and input interfaces, are now encoder-decoder names, with the same interface IDs. The Python projection no longer includes whisper_bindings, transcribe_whisper_log_mel, or WhisperAutomaticSpeechRecognitionBindings. Code that implements IWinMLAutomaticSpeechRecognitionConfiguration must be rebuilt, because the interface gained language, mode, and prediction methods.

Transcribe from a checkpoint folder

Load the speech model's checkpoint folder with LoadModelFromPath, create the Task with the model's tokenizer, and call SetModel with a GPU execution target. The examples use Whisper tiny; point them at another supported speech model folder, such as Nemotron, to use that model. Unlike text generation, speech recognition takes the tokenizer explicitly. Set the language and mode, then create a session:

// Temporary: WinMLRuntime and the other wrapper types in the WinMLSamples.Runtime
// namespaces aren't in the Windows ML package yet. Copy the wrapper files from
// https://github.com/microsoft/WindowsML/tree/main/Samples/Runtime/shared/csharp
// into your project. A future update will include the C# APIs natively.
using WinMLSamples.Runtime;
using WinMLSamples.Runtime.Tasks;

using WinMLRuntime runtime = WinMLRuntime.Create();
using ExecutionTarget target = runtime.CreateExecutionTarget(
    ExecutionTargetKind.Gpu, ExecutionTargetPreference.Performance);
using Model model = runtime.LoadModelFromPath(@"C:\models\whisper-tiny");
using Tokenizer tokenizer = Tokenizer.CreateFromFile(@"C:\models\whisper-tiny\tokenizer.json");

using WinMLTasks tasks = WinMLTasks.Create(runtime);
using AutomaticSpeechRecognitionTask task =
    tasks.AutomaticSpeechRecognition.CreateAutomaticSpeechRecognitionTask(tokenizer);

// Bind the model to the configuration on the GPU target.
using SpeechRecognitionConfiguration configuration = task.CreateConfiguration();
using (ModelSpeechRecognitionConfiguration modelConfiguration =
    configuration.As<ModelSpeechRecognitionConfiguration>())
{
    modelConfiguration.SetModel(target, model);
}

// Without a language, the model's default applies.
configuration.Language = "en";
configuration.Mode = SpeechRecognitionMode.Transcribe;
configuration.Validate();

// A SpeechRecognizer keeps the session on a thread of its own and streams the transcript.
await using SpeechRecognizer recognizer =
    await SpeechRecognizer.CreateAsync(tasks, task, configuration);

To stream a transcript from either path, call TranscribeWaveform as the next section shows, with the C# SpeechRecognizer streaming SpeechRecognitionUpdate values with await foreach.

Transcribe audio with a Whisper pipeline

The following example uses the Whisper encoder-decoder composition. Supply a Runtime tensor holding waveform samples. Load the Whisper encoder and decoder models, build a pipeline for each, and compose them with a tokenizer into a session:

// Temporary: WinMLRuntime and the other wrapper types in the WinMLSamples.Runtime
// namespaces aren't in the Windows ML package yet. Copy the wrapper files from
// https://github.com/microsoft/WindowsML/tree/main/Samples/Runtime/shared/csharp
// into your project. A future update will include the C# APIs natively.
using WinMLSamples.Runtime;
using WinMLSamples.Runtime.Tasks;

// Create the runtime and an execution target (CPU here; GPU/NPU also available).
using WinMLRuntime runtime = WinMLRuntime.Create();
using ExecutionTarget target = runtime.CreateCpuExecutionTarget();

// Load the Whisper encoder as its own single-stage pipeline.
using Model encoderModel = runtime.LoadModelFromFile(@"C:\models\whisper\encoder_model.onnx");
using PipelineBuilder encoderBuilder = runtime.CreatePipelineBuilder();
using Stage encoderStage = encoderBuilder.AddModelStage(encoderModel, target, "whisper-encoder");
using Pipeline encoderPipeline = encoderBuilder.Build();

// Load the Whisper decoder the same way.
using Model decoderModel = runtime.LoadModelFromFile(@"C:\models\whisper\decoder_model.onnx");
using PipelineBuilder decoderBuilder = runtime.CreatePipelineBuilder();
using Stage decoderStage = decoderBuilder.AddModelStage(decoderModel, target, "whisper-decoder");
using Pipeline decoderPipeline = decoderBuilder.Build();

using Tokenizer tokenizer = Tokenizer.CreateFromFile(@"C:\models\whisper\tokenizer.json");

// Compose the encoder/decoder pipelines and tokenizer into a session.
using WinMLTasks tasks = WinMLTasks.Create(runtime);
using AutomaticSpeechRecognitionTask task =
    tasks.AutomaticSpeechRecognition.CreateAutomaticSpeechRecognitionTask(tokenizer);
using SpeechRecognitionConfiguration configuration = task.CreateConfiguration();
using (EncoderDecoderSpeechRecognitionConfiguration whisper =
    configuration.As<EncoderDecoderSpeechRecognitionConfiguration>())
{
    whisper.SetBindings(new EncoderDecoderSpeechRecognitionBindings
    {
        EncoderPipeline = encoderPipeline,
        EncoderStage = encoderStage,
        DecoderPipeline = decoderPipeline,
        DecoderStage = decoderStage,
        TensorTarget = target,
    });
}

configuration.Validate();

// A SpeechRecognizer keeps the session on a thread of its own and streams the transcript.
await using SpeechRecognizer recognizer =
    await SpeechRecognizer.CreateAsync(tasks, task, configuration);

Call TranscribeWaveform to start a streaming operation, then pull fragments from the returned stream until it reports completion:

// Cancelling the token stops transcription in the Runtime.
CancellationTokenSource cancellation = new();

// Read transcript text one fragment at a time until the stream completes.
await foreach (SpeechRecognitionUpdate update in
    recognizer.TranscribeWaveformAsync(waveform, waveformMetadata, cancellation.Token))
{
    Console.Write(update.Text);
}

// The outcome is the authoritative transcript and finish reason.
SpeechRecognitionOutcome outcome = recognizer.LastOutcome!;
Console.WriteLine($"\nFinished because: {outcome.FinishReason}");

// Closing the session reports a failure that disposing it would not.
await recognizer.CloseAsync();

waveform and waveformMetadata come from your own audio-loading code: a Runtime tensor holding mono floating-point PCM samples, plus the sample rate and valid-length metadata that describes it. In C#, target.AudioTensorFactory creates audio tensors.

The completed result reports the full transcript and a finish reason, such as an end-of-transcript token, cancellation, or an error.

C#, C++, and Python

The native API is exposed through WinMLTasks.h and the typed C++ helpers under winml/tasks/speech/asr/WhisperAutomaticSpeechRecognition.hpp. Python applications use the windowsml.tasks projection together with windowsml.runtime. C# applications use CsWin32 bindings through a wrapper that you copy into your project for now; see Set up a project.

The Windows ML samples include C++, C#, and Python speech recognition samples for both ways of specifying a model. The samples show the full Runtime setup beside the speech-recognition configuration so you can see where the abstraction begins and how to customize the underlying workflow.

See also