Analytics Consumption Zone concepts

Native analytics capabilities in Azure Data Manager for Energy, delivered through an analytics consumption zone (ACZ), make energy data readily accessible for analytics and AI. ADME synchronizes selected OSDU® data into customer-owned storage in open Delta Parquet format, enabling use with Microsoft Fabric, Azure Databricks, Snowflake, and other compatible analytics platforms without the need to build and maintain custom data pipelines.

Analytics Consumption Zone (ACZ) exports selected entity data from Azure Data Manager for Energy to your Azure Data Lake Storage Gen2 account. ACZ writes Azure Data Manager for Energy data in open Delta Parquet format. Services like Microsoft Fabric and Azure Databricks can read this format directly.

To enable ACZ, see Enable Analytics Consumption Zone.

What is ACZ?

ACZ is a managed sync layer. It exports entity data from your Azure Data Manager for Energy instance to an Azure Data Lake Storage Gen2 storage account that you own. You can then connect that data to analytics, reporting, and machine learning tools.

Setting up ACZ has two parts. You enable ACZ once on an Azure Data Manager for Energy resource in the Azure portal. Then you create one or more ACZ instances by using the ACZ APIs. Each ACZ instance is an export configuration in a data partition that defines which kinds ACZ exports and where it writes them. ACZ exports data only for the ACZ instances that you create.

Key characteristics of ACZ:

• Customer-owned storage: You create and manage a Data Lake Storage Gen2 storage account where your data goes. We recommend a storage account in the same region as your Azure Data Manager for Energy instance. If you have data residency requirements, you're responsible for selecting a storage account in the appropriate region.
• Open format: Your data exports in Delta Parquet format. Analytics engines widely support this format.
• Selective sync: You choose which entity types to sync. Options include catalog kinds and Wellbore Domain Data Management Service (DDMS) kinds (preview).
• Historical and incremental sync: You get an initial snapshot of existing data from ACZ. Then ACZ synchronizes changes as they occur.
• Portal and API management: You enable ACZ in the Azure portal, and you create and manage ACZ instances by using REST APIs.

Architecture

The following diagram shows the ACZ data flow.

How ACZ works

Supported entity types

ACZ synchronizes two categories of Azure Data Manager for Energy entity types.

ACZ can also export OSDU entitlement membership data through entitlements export (preview).

When you create an ACZ instance, you specify which entity types to synchronize by providing:

• catalogKinds: A list of catalog kind patterns (for example, osdu:wks:master-data--Well:*).
• wellboreDDMSKinds: A list of Wellbore DDMS kind patterns (for example, osdu:wks:work-product-component--WellLog:*).

These kind patterns act as filters that determine which Azure Data Manager for Energy records ACZ exports and keeps synchronized.

Use the allCatalogSync flag

The allCatalogSync flag is an optional Boolean parameter that you can specify when you create an ACZ instance. When set to true, it synchronizes all catalog kinds from the data partition.

Key behaviors:

• allCatalogSync is specified outside the configuration section in the request body.
• When allCatalogSync: true, ACZ exports all catalog kinds automatically.
• The catalogKinds and wellboreDDMSKinds arrays in the configuration are ignored for catalog data.
• Wellbore DDMS bulk file downloads are not affected by this flag. Files are only downloaded for kinds explicitly listed in wellboreDDMSKinds.

Example configurations:

// Selective catalog sync - only Wells and Fields
{
  "allCatalogSync": false,
  "configuration": {
    "catalogKinds": [
      "osdu:wks:master-data--Well:*",
      "osdu:wks:master-data--Field:*"
    ]
  }
}

// Sync all catalog kinds using allCatalogSync flag
{
  "allCatalogSync": true,
  "configuration": {
    // catalogKinds is ignored when allCatalogSync is true
  }
}

// Sync all catalog kinds, but Wellbore DDMS files only for specified kinds
{
  "allCatalogSync": true,
  "configuration": {
    "wellboreDDMSKinds": [
      "osdu:wks:work-product-component--WellLog:*"
    ]
  }
}

Version types

When you create an ACZ instance, you choose how to handle entity versions.

Lifecycle states

Each ACZ goes through these states:

Historical snapshot

When you create a new ACZ instance, the service takes a historical snapshot. This snapshot exports all existing records that match the configured entity types (catalogKinds and wellboreDDMSKinds). The snapshot progresses through the following states:

After the snapshot finishes, ACZ switches to incremental mode. It captures new and updated records in near real time.

How ACZ handles data changes

ACZ propagates created, updated, and deleted records from Azure Data Manager for Energy to the Delta tables.

• Creations and updates: When you create a record or change its data block, Azure Data Manager for Energy creates a new version. ACZ detects the change and writes a new row to the Delta table.
• Metadata-only updates: When a PATCH operation changes the access control list, legal, or tags without creating a new version, ACZ updates the row for the existing version.
• Soft deletes: When you soft-delete a record in Azure Data Manager for Energy, ACZ sets the isActive field to False instead of removing the row. For ACZ instances that use ALL_VERSIONS, all stored versions of the record are marked inactive. Soft deletes preserve history for auditing and time-travel queries.
• Purges: When you purge a record in Azure Data Manager for Energy, ACZ permanently removes the record from the Delta table. The row is deleted and can't be recovered from the ACZ data.

Identify new, updated, and deleted records

ACZ doesn't turn on Delta Lake change data feed on its tables. If you turn it on yourself, each record update appears as a delete followed by an insert. To process changes incrementally, use ingestTime as a watermark. ACZ refreshes ingestTime whenever it writes a change to a row, including:

• New records and new record versions.
• Metadata-only updates, such as changes to tags, that don't create a new record version.
• Soft deletes. ACZ sets isActive to False.

Purged records are removed from the Delta table. To remove purged records from downstream tables, periodically reconcile them against the current ACZ table or rebuild them.

For a reference implementation, see the ADME ACZ Silver Layer sample notebook.

Data output format

ACZ writes data in Delta Lake format with Parquet-encoded files (DELTA_PARQUET). Delta Lake supports atomicity, consistency, isolation, and durability transactions. It also supports time travel and efficient incremental reads.

Data Lake Storage Gen2 folder structure

ACZ writes data to the storage account that you specify when you create an ACZ instance. The first segment of the optional basePath property is the container, and any additional segments are folders in that container. The first segment must be a valid container name. For naming rules, see Naming and referencing containers, blobs, and metadata. If you don't specify a base path, ACZ uses a container named <aczId>. ACZ creates the container if it doesn't exist and writes its data to a folder named <aczId>. For examples, see Create an ACZ instance.

<storage-account>/
└── <container>/                      ← first segment of basePath, or <aczId>
    └── [<more basePath folders>/]
        └── <aczId>/
            ├── osducatalog/          ← Delta table, partitioned by kind
            │   └── _delta_log/
            ├── wellboreDDMS/         ← Wellbore DDMS bulk files (Parquet and JSON)
            │   ├── work-product-component--WellLog/
            │   └── work-product-component--WellboreTrajectory/
            └── entitlements/         ← Delta table (preview), only when entitlements export is enabled

For more information about the entitlements table, see Entitlements export in Analytics Consumption Zone (preview).

Key details

ACZ regularly runs maintenance on its Delta tables, including VACUUM, which removes older data files. Time-travel queries can read only recent table versions. To keep a longer history, copy the data to your own storage on a schedule. If you have specific requirements for table history, create a support request.

Delta table schema

The Delta table has the following fields:

Schema versions

The catalogSchemaVersion property in the ACZ API response shows which schema an ACZ instance uses:

Existing ACZ instances keep their schema version. To use the current schema, create a new ACZ instance. There's no migration from version 1 to version 2.

Entitlements export (preview)

ACZ can optionally export OSDU entitlement membership data to an entitlements Delta Lake table alongside the catalog data. You can use this data to implement entitlement-aware filtering in downstream analytics platforms. ACZ doesn't configure or enforce downstream row-level security.

Entitlements export is disabled by default. To enable it, create a support request. For details, see Entitlements export in Analytics Consumption Zone (preview).

Limits and access

Limits

Authentication and authorization

ACZ requires:

• API access: To call ACZ APIs, you must belong to the users@{data-partition-id}.dataservices.energy, users.datalake.ops@{data-partition-id}.dataservices.energy, and service.entitlements.user@{data-partition-id}.dataservices.energy groups.
• Storage access: The managed identity needs the Storage Blob Data Contributor role (or equivalent) on the Data Lake Storage Gen2 container.
• Azure Data Manager for Energy access: The managed identity is assigned to the Azure Data Manager for Energy resource when you enable ACZ in the Azure portal. For steps, see Enable Analytics Consumption Zone.

Backup and restore

ACZ isn't included in Azure Data Manager for Energy backup and restore. If you restore an Azure Data Manager for Energy resource or data partition from a backup, ACZ data in your storage account no longer matches the restored data. Create new ACZ instances after the restore finishes, which take a new historical snapshot.

Business continuity and disaster recovery

ACZ isn't included in Azure Data Manager for Energy cross-region disaster recovery. After a regional failover, ACZ might not resume synchronization or might be missing changes from the failover window. A historical snapshot that's in progress during a failover might not complete. To restore ACZ, create new ACZ instances, which take a new historical snapshot.

To prepare:

• Configure redundancy for your destination Data Lake Storage Gen2 account. ACZ writes to the storage account that you specify and doesn't replicate it. For options, see Azure Storage redundancy.
• Keep the ACZ configurations, such as kinds, basePath, and storage account, that you need to recreate your ACZ instances.

For more information, see Reliability in Azure Data Manager for Energy.

Related content

• Enable Analytics Consumption Zone
• Tutorial: Use ACZ APIs
• Connect ACZ data to Fabric
• Connect ACZ data to Azure Databricks
• Entitlements export in Analytics Consumption Zone (preview)