Read OpenSharing shares from an Iceberg client using Open ID Connect (OIDC) federation

Open ID Connect (OIDC) federation data recipients can use an Iceberg REST Catalog (IRC) client, such as OSS Spark or Snowflake, to access OpenSharing shares created in Azure Databricks.

If you are using bearer tokens to manage authentication instead, see Create a recipient object for non-Databricks users using bearer tokens (Databricks-to-Open sharing).

This page is intended for recipients. For information about how providers can enable OIDC federation for recipients in Azure Databricks, see Enable Open ID Connect (OIDC) federation for OpenSharing recipients.

Download Iceberg profile for OAuth connectors

  1. Go to the OIDC profile portal URL that the Azure Databricks provider shared with you.

    Request the URL if you haven't yet received it.

  2. On the portal page, select the share you want to access from the drop-down menu.

  3. Click Download file.

  4. In the downloaded file, replace clientId, clientSecret, and scope with your values.

  5. Use the selected share name as the warehouse in your configuration.

  6. Import the share profile file with your client code.

Access the shared data

Access the data using any Iceberg REST Catalog.

Snowflake

In your Snowflake worksheet, run the following command to create a catalog integration. You can also copy the SQL command from the OIDC profile portal your provider shared with you. Make sure to select the share you want to access.

Replace the placeholder values before running it:

  • <catalog-integration-name>: A name for the catalog integration.
  • <catalog-uri>: The Iceberg REST Catalog endpoint. Find it in the OIDC profile portal or your downloaded profile file.
  • <share-name>: The name of the share you selected in the OIDC profile portal.
  • <oauth-token-uri>: Your IdP's OAuth token endpoint.
  • <oauth-client-id> and <oauth-client-secret>: Your OAuth credentials.
  • <oauth-scope>: Your OAuth scope.
USE ROLE ACCOUNTADMIN;

CREATE OR REPLACE CATALOG INTEGRATION <catalog-integration-name>
    CATALOG_SOURCE = ICEBERG_REST
    TABLE_FORMAT = ICEBERG
    REST_CONFIG = (
        CATALOG_URI = '<catalog-uri>'
        WAREHOUSE = '<share-name>'
        ACCESS_DELEGATION_MODE = VENDED_CREDENTIALS
    )
    REST_AUTHENTICATION = (
        TYPE = OAUTH
        OAUTH_TOKEN_URI = '<oauth-token-uri>'
        OAUTH_CLIENT_ID = '<oauth-client-id>'
        OAUTH_CLIENT_SECRET = '<oauth-client-secret>'
        OAUTH_ALLOWED_SCOPES = ('<oauth-scope>')
    )
    REFRESH_INTERVAL_SECONDS = 30 -- Optional
    ENABLED = TRUE;

Create a linked database that uses the catalog integration. Replace <database-name> with a name for the database.

CREATE DATABASE <database-name>
    LINKED_CATALOG = (
        CATALOG = <catalog-integration-name>
    );

OSS Spark

The following example reads shared data from an Iceberg REST Catalog using OSS Spark. Run it from a Spark installation that includes the Iceberg Spark runtime and AWS bundle JARs.

Replace the placeholder values before running it:

  • <spark-home>: The path to your Spark installation.
  • <iceberg-jars-path>: The path to your Iceberg JAR files.
  • <catalog-uri>: The Iceberg REST Catalog endpoint. Find it in the OIDC profile portal or your downloaded profile file.
  • <share-name>: The name of the share you selected in the OIDC profile portal.
  • <oauth-token-uri>: Your IdP's OAuth token endpoint.
  • <oauth-client-id> and <oauth-client-secret>: Your OAuth credentials.
  • <oauth-scope>: Your OAuth scope.
  • <spark-script>: The path to the Scala script to run.
JAVA_HOME=/usr/lib/jvm/java-17-openjdk-amd64 PATH=/usr/lib/jvm/java-17-openjdk-amd64/bin:$PATH <spark-home>/bin/spark-shell \
  --jars <iceberg-jars-path>/iceberg-spark-runtime-4.0_2.13-1.10.2.jar,<iceberg-jars-path>/iceberg-aws-bundle-1.10.2.jar \
  --conf spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions \
  --conf spark.sql.catalog.databricks=org.apache.iceberg.spark.SparkCatalog \
  --conf spark.sql.catalog.databricks.type=rest \
  --conf spark.sql.catalog.databricks.uri=<catalog-uri> \
  --conf spark.sql.catalog.databricks.warehouse=<share-name> \
  --conf spark.sql.catalog.databricks.rest.auth.type=oauth2 \
  --conf spark.sql.catalog.databricks.oauth2-server-uri=<oauth-token-uri> \
  --conf spark.sql.catalog.databricks.credential=<oauth-client-id>:<oauth-client-secret> \
  --conf spark.sql.catalog.databricks.scope=<oauth-scope> \
  --conf spark.sql.catalog.databricks.io-impl=org.apache.iceberg.aws.s3.S3FileIO \
  < <spark-script>
  • Last updated on