Connect Azure Databricks to a OneLake external location

This page explains how to connect to OneLake using an external location. With the connection, you can create Unity Catalog managed objects (such as tables, volumes, and views) and govern your data through Unity Catalog. If you want to read existing OneLake data from Azure Databricks without writing, see Enable OneLake catalog federation.

To connect to a OneLake path, you must have two Unity Catalog securable objects:

• A storage credential, which specifies a managed identity or service principal that grants access to your OneLake storage.
• An external location, which defines the ID-based ABFSS path to your OneLake storage and references the storage credential.

Requirements

In Azure Databricks:

• Your Azure Databricks workspace must be enabled for Unity Catalog.
• You must use serverless compute or a classic compute resource on Databricks Runtime 18.1 or above.
• You must have CREATE STORAGE CREDENTIAL privilege on the Unity Catalog metastore, and CREATE EXTERNAL LOCATION privilege on both the metastore and the storage credential.

In Microsoft Fabric:

• You must have an active Fabric workspace with capacity assigned.
• The managed identity or service principal must have the Viewer role on the target Fabric workspace and be a member of a OneLake security role on the Azure Databricks storage item.
• A Fabric administrator must enable Users can create Azure Databricks Storage items in Admin Portal > Tenant Settings.
• The target Fabric workspace must have Authenticate with OneLake user-delegated SAS tokens enabled in Workspace settings > Delegated settings > OneLake Settings.

Security model

OneLake security is enabled by default for the Azure Databricks storage item, so the item is locked. Fabric workspace roles (Administrator, Member, Contributor, and Viewer) do not grant access to its data. Only members of the storage item's OneLake security role can read or write it.

To keep your data secure:

• Add only the access connector's managed identity (or your service principal) to the OneLake security role.
• Do not add users, groups, or other service principals. Anyone in the role can read and write the data directly, outside Unity Catalog governance.
• Fabric users should read the data through a mirrored Azure Databricks catalog item, not the storage item directly.

Step 1: Configure identity and access in Azure and Fabric

1. In the Azure portal, create or identify an Access Connector for Azure Databricks with a managed identity, or a service principal.
2. In Microsoft Fabric, go to your target workspace.
3. Click the Workspace settings gear button, then click Manage access.
4. Click Add people or groups and search for your managed identity or service principal.
5. Assign the managed identity or service principal the Viewer role.

Step 2: Provision storage in Fabric and get the path

1. In your Fabric workspace, create a Azure Databricks Storage item.

2. Open Manage OneLake security for the storage item. Create a OneLake security role with ReadWrite permission, and add the managed identity or service principal as its only member.

3. Construct the ID-based ABFSS path:

   abfss://<WorkspaceID>@onelake.dfs.fabric.microsoft.com/<DatabricksStorageID>/Files/
   

   The workspace ID and Azure Databricks storage item ID are available in the browser URL when the item is open in Fabric.

   Note

   The external location requires the ID-based path format (GUIDs), not display names. Name-based paths are rejected at creation time.

   Save this path. You use it in Step 4.

Step 3: Create a Unity Catalog storage credential

1. Sign in to your Unity Catalog-enabled Azure Databricks workspace.

2. In the sidebar, click Catalog.

3. Click , then click Create a credential.

4. Select a Credential Type of Azure Managed Identity.

5. Enter a Storage credential name and an optional comment.

6. Enter the Access Connector ID (the resource ID of the access connector associated with your managed identity).

   The resource ID is in the format:

   /subscriptions/<subscription-id>/resourceGroups/<resource-group>/providers/Microsoft.Databricks/accessConnectors/<connector-name>
   

7. (Optional) If you created the access connector using a user-assigned managed identity, enter the Managed Identity ID.

8. Click Create.

Step 4: Create the OneLake external location

1. In the sidebar, click Catalog.

2. Click , then click Create an external location.

3. Enter an External location name.

4. Under Storage type, select OneLake.

5. Under Storage credential, select the credential you created in Step 3.

6. Under URL, enter the ID-based ABFSS path from Step 2. For example:

   abfss://<WorkspaceID>@onelake.dfs.fabric.microsoft.com/<DatabricksStorageID>/Files/
   

7. Click Create.

8. Click Test connection to verify the external location.

Step 5: Create Unity Catalog objects on the OneLake external location

After you create the external location, you can create a catalog that uses it as managed storage. OneLake stores the data for the managed tables (Delta and Iceberg), volumes, materialized views, and streaming tables that you create in that catalog.

-- Create a catalog with the OneLake external location as managed storage
CREATE CATALOG my_onelake_catalog
MANAGED LOCATION '<onelake_external_location_path>';

-- Create a schema
CREATE SCHEMA my_onelake_catalog.sales;

-- Create a managed table
CREATE TABLE my_onelake_catalog.sales.orders (
  id INT,
  name STRING,
  created_at TIMESTAMP
);

-- Insert data
INSERT INTO my_onelake_catalog.sales.orders
VALUES (1, 'example', current_timestamp());

Replace <onelake_external_location_path> with the external location URL or a sub-path within it.

Limitations

This feature has the following limitations:

• Fabric Spark can't read tables with catalog commits

  Catalog commits are enabled by default for Unity Catalog managed tables in new catalogs and schemas through automatic upgrades. Catalog commits make Unity Catalog the source of truth for table state. Delta clients must get the latest table state from Unity Catalog, which Fabric Spark doesn't do. As a result, Fabric Spark, including Fabric notebooks and Spark job definitions, can't read tables with catalog commits through a mirrored Azure Databricks catalog.

  As a workaround, turn off catalog commits on tables that Fabric Spark workloads need to read. Turning off catalog commits also turns off the features that depend on them for that table. See Turn off catalog commits.

  ALTER TABLE <table-name> DROP FEATURE 'catalogManaged';
  

• Databricks-to-Open sharing isn't supported

  You can't use Databricks-to-Open sharing to share managed tables whose storage is on OneLake. Databricks-to-Open sharing gives recipients pre-signed URLs to the table's data files, and OneLake storage paths don't support pre-signed URLs.

• Fabric doesn't show FILE columns

  Fabric doesn't recognize the FILE type, so it leaves FILE columns out of mirrored Azure Databricks catalogs. Fabric still mirrors the table's other columns, but Fabric users can't get from a row to its file.

Troubleshoot

Additional resources

• Grant other users permission to use the external location. See Manage external locations.
• Define managed storage locations at the catalog or schema level. See Specify a managed storage location in Unity Catalog.
• Read existing OneLake data from Azure Databricks without writing. See Enable OneLake catalog federation.