Edit

Deploy WebAssembly (WASM) modules and graph definitions

Azure IoT Operations data flow graphs support WebAssembly (WASM) modules for custom data processing at the edge. You can deploy custom business logic and data transformations as part of your data flow pipelines.

Important

Currently the only connector that supports graph definitions for custom processing is the HTTP/REST connector.

Important

Currently, the operations experience web UI only supports creating and viewing data flow graph artifacts sourced from Azure Container Registry (ACR) and, for built-in transforms, mcr.microsoft.com. To learn more, see Operations experience web UI only displays data flow graph artifacts sourced from Azure Container Registry (ACR) and mcr.microsoft.com.

Prerequisites

To push your own modules and graphs to a private registry like Azure Container Registry (ACR), you also need:

  • Access to a container registry like ACR to store WASM modules and graphs.
  • The OCI Registry As Storage (ORAS) CLI to push WASM modules to the registry.

The Azure CLI examples in this article use environment variables so that you can set each value once and then copy and paste the commands as-is. If you're using the Azure IoT Operations Codespaces environment from the quickstart, these variables are already set for you and you can skip this step. Otherwise, set the following environment variables in your shell before you run the commands.

The following scripts set the most commonly used environment variables:

Environment variable Description
SUBSCRIPTION_ID The ID of the subscription that contains your Azure IoT Operations instance.
RESOURCE_GROUP The name of the resource group that contains your Azure IoT Operations instance.
AIO_INSTANCE_NAME The name of your Azure IoT Operations instance. To list your instances, run az iot ops list -o table.
CLUSTER_NAME The name of the Azure Arc-enabled Kubernetes cluster that hosts your instance.
LOCATION The Azure region to use for new resources, for example eastus.
SUBSCRIPTION_ID=<subscription-id>
RESOURCE_GROUP=<resource-group-name>
AIO_INSTANCE_NAME=<instance-name>
CLUSTER_NAME=<cluster-name>
LOCATION=<region>

You only need to set the variables that this article uses. This article might use additional environment variables for resource names that you choose. The article explains how to set them where they're introduced.

Overview

WASM modules in Azure IoT Operations data flow graphs and connectors let you process data at the edge with high performance and security. WASM runs in a sandboxed environment and supports Rust and Python.

Use prebuilt modules from a public registry

You can use the prebuilt sample WASM modules and graph definitions that are published to the public GitHub Container Registry (ghcr.io) under azure-samples/explore-iot-operations.

Note

ghcr.io requires an authenticated token exchange before it serves even public artifacts, and the current Azure IoT Operations runtime doesn't perform the anonymous exchange. Configure the public-ghcr endpoint with an artifact pull secret backed by a GitHub personal access token (PAT) with the read:packages scope, rather than anonymous authentication. For the endpoint and secret steps, see Use a public registry.

Available sample artifacts

After you create the public-ghcr registry endpoint, reference it in your data flow graphs by using registryEndpointRef: public-ghcr. Because the registry endpoint host is ghcr.io, include the repository path azure-samples/explore-iot-operations in artifact references. The following sample modules and graph definitions are available:

Artifact Description
azure-samples/explore-iot-operations/graph-simple:1.0.0 Simple temperature conversion graph definition
azure-samples/explore-iot-operations/graph-complex:1.0.0 Multi-sensor processing graph definition
azure-samples/explore-iot-operations/temperature:1.0.0 Temperature conversion module (Fahrenheit to Celsius)
azure-samples/explore-iot-operations/window:1.0.0 Time-based windowing module
azure-samples/explore-iot-operations/snapshot:1.0.0 Image processing and object detection module
azure-samples/explore-iot-operations/format:1.0.0 Image format conversion module
azure-samples/explore-iot-operations/humidity:1.0.0 Humidity data processing module
azure-samples/explore-iot-operations/collection:1.0.0 Multi-sensor data aggregation module
azure-samples/explore-iot-operations/enrichment:1.0.0 Metadata enrichment module
azure-samples/explore-iot-operations/filter:1.0.0 Data filtering module

Note

The public sample graph definitions use module references that include the azure-samples/explore-iot-operations repository path, for example azure-samples/explore-iot-operations/temperature:1.0.0. This path is required because the registry endpoint host is ghcr.io. If you copy the artifacts to your own registry, make sure the module references inside your graph definition match the paths where you push the module artifacts.

To use the simple graph with the public registry, see Example 1: Basic deployment with one WASM module and use public-ghcr as the registry endpoint name.

Use a private registry

If you need to use custom modules or want to host your own copies of the sample modules, set up a private container registry like Azure Container Registry (ACR).

Set up container registry

Azure IoT Operations needs a container registry to pull WASM modules and graph definitions. You can use Azure Container Registry (ACR) or another OCI-compatible registry. To create an ACR instance, see Deploy Azure Container Registry. After the registry exists, create a registry endpoint that points at it - see Create a registry endpoint.

Install ORAS CLI

Use the ORAS CLI to push WASM modules and graph definitions to your container registry. For installation instructions, see Install ORAS.

Pull sample modules from public registry

Use prebuilt sample modules:

# Pull sample modules and graphs
oras pull ghcr.io/azure-samples/explore-iot-operations/graph-simple:1.0.0
oras pull ghcr.io/azure-samples/explore-iot-operations/graph-complex:1.0.0
oras pull ghcr.io/azure-samples/explore-iot-operations/temperature:1.0.0
oras pull ghcr.io/azure-samples/explore-iot-operations/window:1.0.0
oras pull ghcr.io/azure-samples/explore-iot-operations/snapshot:1.0.0
oras pull ghcr.io/azure-samples/explore-iot-operations/format:1.0.0
oras pull ghcr.io/azure-samples/explore-iot-operations/humidity:1.0.0
oras pull ghcr.io/azure-samples/explore-iot-operations/collection:1.0.0
oras pull ghcr.io/azure-samples/explore-iot-operations/enrichment:1.0.0
oras pull ghcr.io/azure-samples/explore-iot-operations/filter:1.0.0

Push modules to your registry

Once you have the sample modules and graphs, push them to your container registry. Set the ACR_NAME environment variable to the name of your Azure Container Registry.

Important

Set the following OCI types for discovery in the operations experience:

Artifact type Required OCI type setting Required layer media type
Graph definition Config media type: application/vnd.microsoft.aio.graph.v1+yaml application/yaml
WASM module Artifact type: application/vnd.module.wasm.content.layer.v1+wasm application/wasm

If you use a CI/CD pipeline or other tooling to copy artifacts between registries, verify that it preserves these media types. Some tools strip or replace artifact metadata during transfer, which causes the artifacts to silently disappear from the operations experience. For more information, see Registry artifact requirements.

Choose an artifact layout

The artifact names you use when you push graphs and modules determine the module references you need inside the graph definition. For background on how the registry endpoint host, artifact path, and module reference relate, see Artifact paths and graph module references.

For the Azure sample graphs, preserve the sample repository path when you copy artifacts to your own registry. The graph definitions reference modules using that path:

<YOUR_ACR_NAME>.azurecr.io/azure-samples/explore-iot-operations/graph-simple:1.0.0
<YOUR_ACR_NAME>.azurecr.io/azure-samples/explore-iot-operations/temperature:1.0.0

Use artifact: azure-samples/explore-iot-operations/graph-simple:1.0.0 in the data flow graph. The graph definition uses module: "azure-samples/explore-iot-operations/temperature:1.0.0".

For your own graphs, you can choose a flat layout:

<YOUR_ACR_NAME>.azurecr.io/graph-simple:1.0.0
<YOUR_ACR_NAME>.azurecr.io/temperature:1.0.0

Use artifact: graph-simple:1.0.0 in the data flow graph and module: "temperature:1.0.0" inside the graph definition.

Or choose your own nested layout:

<YOUR_ACR_NAME>.azurecr.io/factory/graphs/graph-simple:1.0.0
<YOUR_ACR_NAME>.azurecr.io/factory/graphs/temperature:1.0.0

Use artifact: factory/graphs/graph-simple:1.0.0 in the data flow graph and module: "factory/graphs/temperature:1.0.0" inside the graph definition.

To ensure the graphs and modules are visible in the operations experience web UI, add the --config and --artifact-type flags as shown in the following example:

# Log in to your ACR
az acr login --name $ACR_NAME

# Push modules to your registry
oras push $ACR_NAME.azurecr.io/azure-samples/explore-iot-operations/graph-simple:1.0.0 --config /dev/null:application/vnd.microsoft.aio.graph.v1+yaml graph-simple.yaml:application/yaml --disable-path-validation
oras push $ACR_NAME.azurecr.io/azure-samples/explore-iot-operations/graph-complex:1.0.0 --config /dev/null:application/vnd.microsoft.aio.graph.v1+yaml graph-complex.yaml:application/yaml --disable-path-validation
oras push $ACR_NAME.azurecr.io/azure-samples/explore-iot-operations/temperature:1.0.0 --artifact-type application/vnd.module.wasm.content.layer.v1+wasm temperature.wasm:application/wasm
oras push $ACR_NAME.azurecr.io/azure-samples/explore-iot-operations/window:1.0.0 --artifact-type application/vnd.module.wasm.content.layer.v1+wasm window.wasm:application/wasm
oras push $ACR_NAME.azurecr.io/azure-samples/explore-iot-operations/snapshot:1.0.0 --artifact-type application/vnd.module.wasm.content.layer.v1+wasm snapshot.wasm:application/wasm
oras push $ACR_NAME.azurecr.io/azure-samples/explore-iot-operations/format:1.0.0 --artifact-type application/vnd.module.wasm.content.layer.v1+wasm format.wasm:application/wasm
oras push $ACR_NAME.azurecr.io/azure-samples/explore-iot-operations/humidity:1.0.0 --artifact-type application/vnd.module.wasm.content.layer.v1+wasm humidity.wasm:application/wasm
oras push $ACR_NAME.azurecr.io/azure-samples/explore-iot-operations/collection:1.0.0 --artifact-type application/vnd.module.wasm.content.layer.v1+wasm collection.wasm:application/wasm
oras push $ACR_NAME.azurecr.io/azure-samples/explore-iot-operations/enrichment:1.0.0 --artifact-type application/vnd.module.wasm.content.layer.v1+wasm enrichment.wasm:application/wasm
oras push $ACR_NAME.azurecr.io/azure-samples/explore-iot-operations/filter:1.0.0 --artifact-type application/vnd.module.wasm.content.layer.v1+wasm filter.wasm:application/wasm

Tip

You can also push your own modules and create custom graphs, see Configuration of custom data flow graphs.

Update a module in a running graph

You can update a WASM module in a running graph without stopping the graph. This is useful when you want to update the logic of an operator without stopping the dataflow. For example, to update the temperature conversion module from version 1.0.0 to 2.0.0 in the Azure sample artifact layout, upload the new version as follows:

oras push $ACR_NAME.azurecr.io/azure-samples/explore-iot-operations/temperature:2.0.0 --artifact-type application/vnd.module.wasm.content.layer.v1+wasm temperature.wasm:application/wasm

Note

If you push new content to the same tag (for example, overwriting azure-samples/explore-iot-operations/temperature:1.0.0), the data flow graph automatically picks up the updated module without additional configuration. However, if you push to a new tag (for example, azure-samples/explore-iot-operations/temperature:2.0.0), you must also update the graph definition YAML to reference the new version and re-push the graph artifact.

Develop custom WASM modules

To create custom data processing logic for your data flow graphs, develop WebAssembly modules in Rust or Python. Custom modules enable you to implement specialized business logic, data transformations, and analytics that aren't available in the built-in operators.

For comprehensive development guidance including:

  • Setting up your development environment
  • Creating operators in Rust and Python
  • Understanding the data model and interfaces
  • Building and testing your modules

See Develop WebAssembly modules for data flow graphs.

For detailed information about creating and configuring the YAML graph definitions that define your data processing workflows, see Configure WebAssembly graph definitions.

Registry artifact requirements

The operations experience uses OCI artifact metadata to discover and display graphs and modules. Understanding these requirements is important when you build custom CI/CD pipelines, copy artifacts between registries, or troubleshoot missing artifacts in the UI.

How artifact discovery works

ORAS uses the following OCI type metadata:

  • Config media type: Identifies a graph definition. Set it with --config.
  • Artifact type: Identifies a WASM module. Set it with --artifact-type.
  • Layer media type: Describes the content format of the actual file (YAML or WASM).

Preserve this metadata when publishing or copying artifacts.

Required media types

Artifact type OCI type metadata Layer media type
Graph definition config.mediaType: application/vnd.microsoft.aio.graph.v1+yaml application/yaml
WASM module artifactType: application/vnd.module.wasm.content.layer.v1+wasm application/wasm

For graph definitions, pass the config media type with the --config flag. Set the REGISTRY environment variable to your registry host (for example, <your-registry>.azurecr.io):

oras push $REGISTRY/my-graph:1.0.0 \
  --config /dev/null:application/vnd.microsoft.aio.graph.v1+yaml \
  graph.yaml:application/yaml \
  --disable-path-validation

For WASM modules, pass the artifact type with the --artifact-type flag:

oras push $REGISTRY/my-module:1.0.0 \
  --artifact-type application/vnd.module.wasm.content.layer.v1+wasm \
  module.wasm:application/wasm

For OCI 1.1 modules, --artifact-type sets artifactType; config.mediaType can be application/vnd.oci.empty.v1+json.

CI/CD pipeline considerations

When you copy artifacts between registries, preserve the graph's config media type and the module's artifact type. Missing type metadata can hide artifacts in the operations experience.

To verify that an artifact has the correct metadata after transfer, inspect its manifest:

oras manifest fetch $REGISTRY/my-graph:1.0.0 | jq '{mediaType, configMediaType: .config.mediaType}'

The output should show:

{
  "mediaType": "application/vnd.oci.image.manifest.v1+json",
  "configMediaType": "application/vnd.microsoft.aio.graph.v1+yaml"
}

For graph definitions, repush with --config if configMediaType doesn't match the value above.