Enable OneLake catalog federation

This article shows how to read data in OneLake using catalog federation. This allows Unity Catalog queries to run directly against OneLake storage.

OneLake federation enables you to analyze data stored in your Lakehouse or Warehouse without copying it, bringing analytics and AI/BI capabilities in Azure Databricks directly to your OneLake data. Data access is read-only.

Requirements

You must meet the following requirements to run federated queries on OneLake using catalog federation:

Workspace requirements:

• Workspace enabled for Unity Catalog.

Compute requirements:

• Network connectivity from your compute resource to OneLake. If the Fabric workspace blocks public internet access, see Connect to a Fabric workspace that blocks public access. For general guidance, see Networking recommendations for Lakehouse Federation.
• Azure Databricks compute must use Databricks Runtime 18.0 or above and standard access mode.
• SQL warehouses must use 2025.40 or above.

Permissions required:

• To create a connection, you must be a metastore admin or a user with the CREATE CONNECTION and CREATE STORAGE CREDENTIAL privileges on the Unity Catalog metastore attached to the workspace.
• To create a foreign catalog, you must have the CREATE CATALOG permission on the metastore and be either the owner of the connection or have the CREATE FOREIGN CATALOG privilege on the connection.

Additional permission requirements are specified in each task-based section that follows.

• You must have permissions to create resources in Azure, configure access in Fabric, and manage Unity Catalog in Azure Databricks.
• Supported authentication methods:
  • Azure Managed Identity via an Access Connector for Azure Databricks
  • Azure service principal

Fabric tenant and workspace settings:

• A Fabric administrator must enable the Service principals can call Fabric public APIs tenant setting. This allows managed identities and service principals to authenticate with Fabric. For more information, see Service principal tenant setting.
• A Fabric administrator must enable the Allow apps running outside of Fabric to access data via OneLake tenant setting. This allows Azure Databricks to connect to OneLake storage. For more information, see Allow apps running outside of Fabric to access data via OneLake.
• A Fabric administrator must enable the Use short-lived user-delegated SAS tokens tenant setting. This setting allows OneLake to issue the short-lived, Entra-backed SAS tokens that Azure Databricks uses to read OneLake data. For more information, see OneLake shared access signatures.
• In the Fabric workspace that contains your data, Authenticate with OneLake user-delegated SAS tokens must be enabled. Navigate to Workspace settings > Delegated settings > OneLake settings to enable this setting.
• If your Fabric workspace restricts inbound public network access, the setup depends on its tenant. If the Fabric workspace is in the same Microsoft Entra tenant as your Azure Databricks workspace, a Fabric workspace admin must add the resource ID of the Access Connector for Azure Databricks to the workspace's resource instance rules. If it's in another tenant, you connect through a private endpoint. See Connect to a Fabric workspace that blocks public access.

The following Fabric data items are supported:

• Fabric Lakehouse
• Fabric Warehouse

Set up catalog federation

The following steps guide you through creating the connection and foreign catalog for OneLake federation.

Step 1: Set up Azure authentication

OneLake federation supports two authentication methods.

• Azure Managed Identity (recommended): Uses an Access Connector for Azure Databricks with a managed identity.
• Azure Service Principal: Uses a Microsoft Entra ID application with client credentials. This method supports cross-tenant authentication, allowing Azure Databricks to access Fabric workspaces located in a different Azure tenant. See Connect to a Fabric workspace in another tenant.

Option A: Create an access connector (Managed Identity)

The Access Connector for Azure Databricks creates a managed identity that Azure Databricks uses to authenticate with OneLake.

1. In the Azure Portal, search for and create a new Access Connector for Azure Databricks resource.

2. Follow the prompts to create the connector. This resource creates a system-assigned Managed Identity.

3. Record the Resource ID of the newly created connector. You need this ID when creating the Unity Catalog storage credential.

   The resource ID is in the format:

   /subscriptions/<subscription-id>/resourceGroups/<resource-group>/providers/Microsoft.Databricks/accessConnectors/<connector-name>
   

For more information about creating access connectors and using managed identities, see Use Azure managed identities in Unity Catalog to access storage.

Option B: Register a service principal

If you prefer to use a service principal instead of a managed identity:

1. In the Azure Portal, navigate to Microsoft Entra ID > App registrations and register a new application (or use an existing one).
2. Record the Application (client) ID and the Directory (tenant) ID.
3. Under Certificates & secrets, create a new client secret and record the secret value.

If the Fabric workspace is in a different Microsoft Entra tenant, the application must have a service principal in that tenant. See Connect to a Fabric workspace in another tenant.

Step 2: Grant permissions in Fabric

Grant the managed identity or service principal permission to read the Fabric data. You can grant a workspace role, which covers every data item in the workspace, or grant permissions on only the data items you want to federate.

Option A: Assign a workspace role

1. In the Fabric portal, navigate to the workspace that contains your Lakehouse or Warehouse data items.
2. In the workspace, click the Workspace settings gear icon, then click Manage access.
3. Click Add people or groups.
4. Search for and select the Managed Identity or service principal. For Managed Identity, the name should match the access connector you created earlier. For a service principal, search for the application name you registered.
5. Assign the identity the Contributor role at minimum. You can also assign Member or Admin roles.
6. Click Add.
7. Verify that the identity appears in the access list with the appropriate role. Permissions on individual Lakehouse and Warehouse items are inherited from the workspace-level role.

Option B: Grant permissions on individual data items

Azure Databricks reads OneLake data through the OneLake APIs, so the identity needs a permission that grants OneLake read access. The SQL-only permission (Read all with SQL analytics endpoint) is not sufficient.

Before you assign permissions on individual data items, assign the identity the Viewer role at the Fabric workspace level. Follow the steps in Option A, selecting Viewer instead of Contributor.

1. In the Fabric portal, navigate to the workspace that contains your data items.
2. Next to the data item you want to federate, click the ellipsis (...), then click Manage permissions.
3. Click Add user, then search for and select the Managed Identity or service principal.
4. Select the Read all permission that grants OneLake read access. Depending on the item type and your Fabric version, this permission is labeled Read all with Apache Spark or Read all OneLake data. For more information, see Item permissions in the Microsoft Fabric documentation.
5. Click Grant.
6. Repeat these steps for each data item you want to federate.

Step 3: Create a storage credential

Create a storage credential in Unity Catalog that references the identity you configured in Step 1.

1. In your Azure Databricks workspace, click Catalog.
2. At the top of the Catalog pane, click the plus icon and select Create a credential from the menu.

If you're using a Managed Identity:

1. In the Create a new credential modal, for Credential Type, choose Azure Managed Identity.
2. For Credential name, enter a Name for the storage credential (for example, onelake_storage_cred).
3. For Access connector ID, enter the resource ID of the access connector you created earlier.
4. (Optional) Add a comment.
5. Click Create.

If you are using a Service Principal:

You cannot create a service principal storage credential using Catalog Explorer. You must be a Azure Databricks account admin and use the Storage Credentials API. For example:

curl -X POST -n \
https://<databricks-instance>/api/2.1/unity-catalog/storage-credentials \
-d '{
   "name": "<storage-credential-name>",
   "read_only": true,
   "azure_service_principal": {
      "directory_id": "<directory-id>",
      "application_id": "<application-id>",
      "client_secret": "<client-secret>"
   },
   "skip_validation": "false"
   }'

You can also create a storage credential by using Databricks Terraform provider and databricks_storage_credential.

If the Fabric workspace is in a different Microsoft Entra tenant, set directory_id to the Fabric tenant's directory ID. See Connect to a Fabric workspace in another tenant.

Step 4: Create a Unity Catalog connection

Create a Unity Catalog connection that uses the storage credential to access OneLake.

Catalog Explorer

1. In your Azure Databricks workspace, click Catalog.
2. At the top of the Catalog pane, click the plus icon and select Create a connection from the menu.
3. On the Connection basics page, enter a Connection name (for example, onelake_connection).
4. Select a Connection type of OneLake.
5. (Optional) Add a comment.
6. Click Next.
7. On the Connection details page, for Credential, select the storage credential you created in the previous step (for example, onelake_storage_cred).
8. For Workspace, enter the workspace ID of your Fabric workspace.
9. Click Create connection.

SQL

Run the following command in a notebook or the Databricks SQL query editor. Replace the placeholder values:

• <connection-name>: A name for the connection.
• <workspace-id>: The GUID of your OneLake workspace.
• <storage-credential-name>: The name of the storage credential you created in step 3.

CREATE CONNECTION <connection-name> TYPE onelake
OPTIONS (
  workspace '<workspace-id>',
  credential '<storage-credential-name>'
);

Step 5: Create a foreign catalog

A foreign catalog links a specific Fabric data item to a catalog in Unity Catalog.

Get the Fabric data item ID

1. In the Fabric portal, navigate to the target Lakehouse or Warehouse.

2. Copy the Data Item ID, which is a GUID (for example, f089354e-8366-4e18-aea3-4cb4a3a50b48).

   You can find this GUID in the Fabric UI or by copying it from your browser URL when you navigate to the Lakehouse or Warehouse:

   https://app.fabric.microsoft.com/groups/<workspace-id>/lakehouses/<data-item-id>?experience=power-bi
   

Create the catalog

Catalog Explorer

1. In your Databricks workspace, click Catalog.
2. At the top of the Catalog pane, click the plus icon and select Create a catalog from the menu.
3. On the Create a new catalog dialog, enter a name for the catalog (for example, fabric_sales).
4. Select a Type of Foreign.
5. Select the Connection you created in Step 4 (for example, onelake_connection).
6. For Data item, enter the data item ID you copied from the Fabric portal.
7. For Item type, select the type that matches your data item from the dropdown list.
8. (Optional) Click Test connection to validate your configuration.
9. Click Create.

SQL

Run the following SQL command in a notebook or the Databricks SQL query editor. Items in square brackets are optional. Replace the placeholder values:

• <catalog-name>: Name for the catalog in Azure Databricks.
• <connection-name>: The connection you created in step 4.
• <data-item-id>: GUID of the Fabric Lakehouse or Warehouse.
• <item-type>: Lakehouse or Warehouse.

CREATE FOREIGN CATALOG [IF NOT EXISTS] <catalog-name> USING CONNECTION <connection-name>
OPTIONS (
  data_item '<data-item-id>',
  item_type '<item-type>'
);

You can also specify the following optional parameters:

CREATE FOREIGN CATALOG <catalog-name> USING CONNECTION <connection-name>
OPTIONS (
  data_item '<data-item-id>',
  item_type 'Lakehouse',
  create_volume_for_lakehouse_files 'true'
);

• create_volume_for_lakehouse_files: true (default) or false. When true, creates a volume for the /Files folder of the Lakehouse. Applies only to Lakehouse items. See Access unstructured data.

The catalog syncs automatically, making the Fabric tables available immediately.

Connect to a Fabric workspace in another tenant

The Fabric workspace that you federate can be in a different Microsoft Entra tenant than your Azure Databricks workspace. Cross-tenant access requires a service principal in the tenant that contains the Fabric workspace, because managed identities can't authenticate to another tenant.

Complete the steps in Set up catalog federation with these changes:

1. In Step 1, use a Microsoft Entra application that has a service principal in the Fabric tenant. The application can be a single-tenant application registered in the Fabric tenant, or a multi-tenant application that a Microsoft Entra ID administrator in the Fabric tenant has consented to. See Application and service principal objects in Microsoft Entra ID.
2. In Step 2, the Fabric workspace admin grants that service principal access to the workspace or to individual data items. An administrator of the Fabric tenant must also enable the Fabric tenant and workspace settings listed in Requirements. The application doesn't need any API permissions in Microsoft Entra ID.
3. In Step 3, set directory_id to the directory (tenant) ID of the Fabric tenant, not your own tenant. Unity Catalog uses the service principal to request tokens from the tenant that contains the Fabric workspace.

If the Fabric workspace blocks public internet access, set up a private endpoint before you create the connection in Step 4. See Private endpoint for a Fabric workspace in another tenant.

Connect to a Fabric workspace that blocks public access

A Fabric workspace can block public internet access to OneLake. How you connect to it depends on whether the Fabric workspace is in the same Microsoft Entra tenant as your Azure Databricks workspace:

• Same tenant: Allowlist your access connector with a resource instance rule in the Fabric workspace. See Fabric workspace in the same tenant.
• Another tenant: Connect through a private endpoint to the Fabric workspace's Private Link service. See Private endpoint for a Fabric workspace in another tenant.

Fabric workspace in the same tenant

When the Fabric workspace is in the same tenant as your Azure Databricks workspace, authenticate with a managed identity and allow the access connector through a resource instance rule on the Fabric workspace. Resource instance rules don't apply across tenants.

1. In Set up catalog federation, use Option A in Step 1 to create an access connector, and record its resource ID. It has the format /subscriptions/<subscription-id>/resourceGroups/<resource-group>/providers/Microsoft.Databricks/accessConnectors/<connector-name>.
2. Ask the Fabric workspace admin to add the access connector's resource ID as a resource instance rule. In the Fabric workspace, the admin selects Workspace settings > Inbound networking, restricts access to selected networks and approved resources, and adds the resource ID. A Fabric administrator must first enable the tenant setting that allows workspace-level inbound network protection. See Manage inbound access to OneLake with Resource Instance Rules in the Microsoft Fabric documentation.
3. Complete the remaining steps in Set up catalog federation.

Private endpoint for a Fabric workspace in another tenant

When the Fabric workspace is in another tenant, Azure Databricks compute connects to OneLake through a private endpoint to the workspace's Private Link service, using a workspace-level private link. Follow Cross-tenant connections using workspace-level private links in the Microsoft Fabric documentation. An administrator of the Fabric tenant creates the Private Link service, and you create the private endpoint as described in the following sections.

Requirements:

• Classic compute must use Databricks Runtime 19 or above.
• You need the resource ID of the Fabric workspace's Private Link service, which the Fabric workspace admin can give you. It has the format /subscriptions/<subscription-id>/resourceGroups/<resource-group>/providers/Microsoft.Fabric/privateLinkServicesForFabric/<name>.

Each Fabric workspace has its own Private Link service, so you create one private endpoint for each Fabric workspace that you federate. Create the private endpoint before you create the connection and foreign catalog in Set up catalog federation.

Over a private endpoint, OneLake federation supports tables in Lakehouse and Warehouse items. Access to Lakehouse files isn't supported. See Access unstructured data.

Serverless compute

Databricks recommends serverless compute for this setup, because Azure Databricks creates the private endpoint and manages its DNS for you.

1. Create a network connectivity configuration (NCC) and attach it to your workspace. See Create a network connectivity configuration and Attach an NCC to a workspace.
2. Add a private endpoint rule to the NCC. For Azure resource ID, enter the Private Link service resource ID. For Azure subresource type, enter workspace. See Create private endpoint rules.
3. Ask an administrator in the Fabric tenant to approve the pending private endpoint connection.
4. Refresh the NCC page until the rule status is ESTABLISHED, and then restart any running serverless compute. See Restart serverless compute plane resources and test the connection.

Classic compute

Classic compute requires a workspace deployed in your own virtual network. See Deploy Azure Databricks in your Azure virtual network (VNet injection).

The following steps adapt Steps 5 through 7 of Cross-tenant connections using workspace-level private links in the Microsoft Fabric documentation. Use the workspace's virtual network instead of creating a new virtual network and virtual machine.

1. In the Azure portal, create a private endpoint in a subnet of the workspace's virtual network. Select Connect to an Azure resource by resource ID or alias, enter the Private Link service resource ID, and select workspace as the target subresource. The Microsoft.Fabric resource provider must be registered in the Azure subscription where you create the private endpoint.
2. Ask an administrator in the Fabric tenant to approve the pending private endpoint connection.
3. Create a private DNS zone named privatelink.fabric.microsoft.com and link it to the workspace's virtual network. Make sure that every record listed on the private endpoint's DNS configuration page is included in this DNS zone, including the OneLake endpoints such as <workspace-id>.z<xy>.dfs.fabric.microsoft.com. In these names, <workspace-id> is the Fabric workspace ID without dashes and <xy> is its first two characters.
4. If network security group rules restrict outbound traffic from the workspace subnets, allow port 443 to the private endpoint.

Verify private connectivity

After you create the foreign catalog, query a table in it. A successful query confirms that traffic uses the private endpoint.

You can also test the connection and browse the foreign catalog's schemas and tables in Catalog Explorer.

Access unstructured data

OneLake federation supports read-only access to unstructured data stored in the /Files folder of a Lakehouse. This is achieved through Unity Catalog volumes.

When you create a foreign catalog with create_volume_for_lakehouse_files set to true (the default), a volume is automatically created under the onelake-folders schema. This volume points to the /Files folder of the Lakehouse and provides read-only access to files stored there.

You can browse and read files using the volume path:

LIST '/Volumes/<catalog-name>/onelake-folders/files/';

df = spark.read.format("binaryFile").load("/Volumes/<catalog-name>/onelake-folders/files/")

You can also query structured files directly:

SELECT * FROM parquet.`/Volumes/<catalog-name>/onelake-folders/files/table.parquet`

Grant permissions on federated tables

After setting up catalog federation, users must have the appropriate Unity Catalog permissions to access federated tables:

• All users need USE CATALOG and USE SCHEMA permissions on the catalog and schema respectively.
• To read from the federated table, users need the SELECT permission.

For more information about Unity Catalog privileges and how to grant them, see Manage privileges in Unity Catalog.

Query OneLake data

After the setup is complete, you can find and query OneLake data in Unity Catalog.

Browse the catalog

1. In your Databricks workspace, navigate to Catalog Explorer.
2. Locate the catalog you created (for example, fabric_sales).
3. Expand the catalog to see the synchronized schemas and tables from the Fabric Lakehouse or Warehouse.

Run queries

Use the three-part naming convention (catalog.schema.table) in Databricks SQL or notebooks:

SELECT COUNT(*)
FROM fabric_sales.silver.customer_details;

SELECT
  customer_id,
  customer_name,
  total_purchases
FROM fabric_sales.silver.customer_details
WHERE total_purchases > 1000
ORDER BY total_purchases DESC
LIMIT 10;

Limitations

OneLake federation has the following limitations:

• Read-only access: Only SELECT queries are supported. Write operations are not available.
• Authentication: Azure Managed Identity and Azure service principal are the supported authentication methods.
• Supported data items: Only Fabric Lakehouse and Warehouse items are supported.
• Compute requirements: Dedicated access mode is not supported.
• Complex datatypes (arrays, maps, structs) are not supported.
• Materialized views and views are not supported.
• You cannot update connection options (such as workspace) after creating the connection.
• Tables with columns that differ only in case (for example, id and ID) are not supported.
• Lakehouse files are not supported for Fabric workspaces in another tenant that you connect to through a private endpoint. See Access unstructured data.