Workspace secret scopes

A workspace secret scope is a named collection of secrets in one workspace. Secret scope permissions control who can read, write, and manage the secrets in it. Permissions apply to the whole scope, not to individual secrets, so a user who can read one secret in a scope can read all of them. To compare workspace secret scopes with Unity Catalog secrets, see Choose where to store a secret.

To use a workspace secret scope:

1. Create a secret scope.
2. Add secrets to the scope.
3. Grant permissions on the scope.
4. Reference the secrets in your code, or in a Spark configuration property or environment variable. See Use a secret in a Spark configuration property or environment variable.

Manage secret scopes

Azure Databricks recommends aligning secret scopes to roles or applications rather than individuals.

There are two types of secret scope:

• Azure Key Vault-backed: You can reference secrets stored in an Azure Key Vault using Azure Key Vault-backed secret scopes. An Azure Key Vault-backed secret scope is a read-only interface to the Key Vault. You manage its secrets in Azure.
• Databricks-backed: A Databricks-backed secret scope is stored in an encrypted database owned and managed by Azure Databricks.

After creating a secret scope, you can assign permissions to grant users access to read, write, and manage secret scopes.

Create an Azure Key Vault-backed secret scope

Create an Azure Key Vault-backed secret scope in the Azure portal and the Azure Databricks workspace UI. You can also use the Databricks CLI.

Requirements

• You must have an Azure key vault instance. If you do not have a key vault instance, follow the instructions in Create a Key Vault using the Azure portal.
• You must have the Key Vault Contributor, Contributor, or Owner role on the Azure key vault instance that you want to use to back the secret scope.

Unable to grant read/list permission to Databricks service principal to KeyVault 'https://xxxxx.vault.azure.net/': Status code 403, {"odata.error":{"code":"Authorization_RequestDenied","message":{"lang":"en","value":"Insufficient privileges to complete the operation."},"requestId":"XXXXX","date":"YYYY-MM-DDTHH:MM:SS"}}

Configure your Azure key vault instance for Azure Databricks

1. Sign in to the Azure portal, find and select the Azure key vault instance.

2. Under Settings, click the Access configuration tab.

3. Set Permission model to Vault access policy. New key vaults default to Azure role-based access control, which Azure Key Vault-backed secret scopes do not support.

   Note

   Creating an Azure Key Vault-backed secret scope grants the Get and List permissions to the application ID for the Azure Databricks service using Vault access policies.

4. Under Settings, select Networking.

5. In Firewalls and virtual networks set Allow access from: to Allow public access from specific virtual networks and IP addresses.

   Under Exception, check Allow trusted Microsoft services to bypass this firewall.

   Note

   You can also set Allow access from: to Allow public access from all networks.

Create an Azure Key Vault-backed secret scope

1. Go to https://<databricks-instance>#secrets/createScope. Replace <databricks-instance> with the workspace URL of your Azure Databricks deployment. This URL is case sensitive. For example, scope in createScope must use an uppercase S.

   

2. Enter the name of the secret scope. Secret scope names are case insensitive.

3. In Manage Principal select Creator or All workspace users to specify which users have the MANAGE permission on the secret scope.

   The MANAGE permission allows users to read, write, and grant permissions on the scope. Your account must have the Premium plan to choose Creator.

4. Enter the DNS Name (for example, https://databrickskv.vault.azure.net/) and Resource ID, for example:

   /subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourcegroups/databricks-rg/providers/Microsoft.KeyVault/vaults/databricksKV
   

   These properties are available from the Settings > Properties tab of an Azure Key Vault in your Azure portal.

5. Click Create.

6. Use the Databricks CLI databricks secrets list-scopes command to verify that the scope was created successfully.

Create a Databricks-backed secret scope

Create a secret scope with the Databricks CLI (version 0.205 and above) or the Secrets API.

Secret scope names:

• Must be unique within a workspace.
• Must consist of alphanumeric characters, dashes, underscores, @, and periods, and cannot exceed 128 characters.
• Are case insensitive.

Secret scope names are considered non-sensitive and are readable by all users in the workspace.

To create a scope using the Databricks CLI:

databricks secrets create-scope <scope-name>

By default, scopes are created with MANAGE permission for the user who created the scope. After you have created a Databricks-backed secret scope, you can add secrets to it.

List secret scopes

To list the existing scopes in a workspace using the CLI:

databricks secrets list-scopes

You can also list secret scopes using the Secrets API.

Delete a secret scope

Deleting a secret scope deletes all secrets and ACLs applied to the scope. To delete a scope using the CLI, run the following:

databricks secrets delete-scope <scope-name>

You can also delete a secret scope using the Secrets API.

Manage secrets

A secret is a key-value pair that stores sensitive material using a key name that is unique within a secret scope.

The method for creating a secret depends on whether you are using an Azure Key Vault-backed scope or a Databricks-backed scope.

Create a secret in an Azure Key Vault-backed scope

To create a secret in Azure Key Vault you use the Azure portal or Azure Set Secret REST API.

Create a secret in a Databricks-backed scope

Create a secret with the Databricks CLI (version 0.205 and above), the Databricks SDK for Python in a notebook, or the Secrets API. Secret names are case insensitive.

Databricks CLI

When you create a secret in a Databricks-backed scope, you can specify the secret value in one of three ways:

• Specify the value as a string with the --string-value flag.
• Input the secret when prompted interactively (single-line secrets).
• Pass the secret using standard input (multi-line secrets).

For example:

databricks secrets put-secret --json '{
  "scope": "<scope-name>",
  "key": "<key-name>",
  "string_value": "<secret>"
}'

If you are creating a multi-line secret, you can pass the secret using standard input. For example:

(cat << EOF
this
is
a
multi
line
secret
EOF
) | databricks secrets put-secret <scope-name> <key-name>

Databricks SDK for Python

from databricks.sdk import WorkspaceClient

w = WorkspaceClient()

w.secrets.put_secret("<secret_scope>","<key-name>",string_value ="<secret>")

Read a secret

Read a secret with the Databricks CLI (version 0.205 and above), or in a notebook with Secrets utility (dbutils.secrets).

Databricks CLI

To read the value of a secret using the Databricks CLI, you must decode the base64 encoded value. You can use jq to extract the value and base64 --decode to decode it:

databricks secrets get-secret <scope-name> <key-name> | jq -r .value | base64 --decode

Secrets utility (dbutils.secrets)

password = dbutils.secrets.get(scope = "<scope-name>", key = "<key-name>")

List secrets

To list secrets in a given scope:

databricks secrets list-secrets <scope-name>

The response shows metadata about the secrets, such as their key names. To list this metadata in a notebook or job, use Secrets utility (dbutils.secrets). For example:

dbutils.secrets.list('my-scope')

Delete a secret

To delete a secret from a scope with the Databricks CLI:

databricks secrets delete-secret <scope-name> <key-name>

You can also use the Secrets API.

To delete a secret from a scope backed by Azure Key Vault, use the Azure SetSecret REST API or Azure portal UI.

Manage secret scope permissions

By default, the user who creates a secret scope has the MANAGE permission on it, which lets them read secrets in the scope, write secrets to it, and manage its permissions.

Manage secret scope permissions with the Databricks CLI (version 0.205 and above) or the Secrets API. For secret permission levels, see Secret ACLs.

Grant a user permissions on a secret scope

To grant a user permissions on a secret scope using the Databricks CLI:

databricks secrets put-acl <scope-name> <principal> <permission>

Making a put request for a principal that already has an applied permission overwrites the existing permission level.

The principal field specifies an existing Azure Databricks principal. A user is specified using their email address, a service principal using its applicationId value, and a group using its group name. For more information, see Principal.

For example, to let a group such as datascience read the secrets in a scope, grant it READ:

databricks secrets put-acl <scope-name> <group> READ

View secret scope permissions

To view all secret scope permissions for a given secret scope:

databricks secrets list-acls <scope-name>

To get the secret scope permissions applied to a principal for a given secret scope:

databricks secrets get-acl <scope-name> <principal>

If no ACL exists for the given principal and scope, this request fails.

Delete a secret scope permission

To delete a secret scope permission applied to a principal for a given secret scope:

databricks secrets delete-acl <scope-name> <principal>

Secret redaction

Storing credentials as Azure Databricks secrets helps protect them when you run notebooks and jobs. However, you can still accidentally print a secret to standard output buffers or display the value during variable assignment.

To prevent this, Azure Databricks attempts to redact secret values that are read using dbutils.secrets.get() and referenced in a Spark configuration property. When displayed, the secret values are replaced with [REDACTED].

For example, if you set a variable to a secret value using dbutils.secrets.get() and then print that variable, that variable is replaced with [REDACTED].

Secret redaction in SQL

Azure Databricks attempts to redact all SQL data query language (DQL) commands that invoke the secret function, including referenced views and user-defined functions. When the secret function is used, the output is replaced with [REDACTED] where possible. Like notebook redaction, this only applies to literal values, not to transformed or indirectly referenced secrets.

For SQL data manipulation language (DML) commands, Azure Databricks allows secret lookups if the secret is considered safe. For example, a secret is safe when wrapped in a cryptographic function such as sha() or aes_encrypt(), which prevent raw values from being stored unencrypted.

Secret validation in SQL

Azure Databricks also applies validation to block SQL DML commands that could result in unencrypted secrets being saved to tables. The query analyzer attempts to identify and prevent these scenarios, which helps avoid accidental storage of sensitive information in plaintext.

Additional resources