Lakehouse Real-Time

Lakehouse Real-Time (Lakehouse//RT) is a serverless compute built for low-latency, high-concurrency use cases, such as serving analytical data to custom applications, running operational analytics, or powering BI dashboards that require sub-second responses for hundreds to thousands of concurrent users.

Lakehouse//RT offers sub-second latency on SQL read queries against your Unity Catalog tables that use Delta Lake or Apache Iceberg formats in cloud storage. You create and manage Lakehouse//RT much like you do other SQL warehouses. A workspace admin or privileged user creates one or more per workspace and assigns permissions to users.

Requirements

To use Lakehouse//RT, you must:

• Be in a supported region.
• Have the Lakehouse//RT Beta enabled in your workspace.

Enable Lakehouse//RT in your workspace

Workspace admins can enable the Lakehouse//RT Beta in your workspace:

1. In your workspace menu (top-right corner), go to Previews.
2. Search for Lakehouse RT.
3. Enable the preview.

After you enable the preview, the Real-Time warehouse type becomes available in the SQL warehouse creation flow for your workspace.

Create a Lakehouse//RT warehouse

To create a Lakehouse//RT warehouse:

1. Go to Compute > SQL Warehouses > Create SQL Warehouse.
2. Select Real-Time.
3. Select a Query size: Small, Medium, Large, or X-Large, depending on the performance your queries require.
4. Configure the Autoscaling to determine the maximum DBUs you can be billed per hour.
5. Set Auto stop to control when an idle warehouse stops.
6. Enter a name for the warehouse.
7. Click Create.

To assign permissions, grant Can use, Can monitor, or Can manage to users and groups, the same as a SQL warehouse.

Size and scale a Lakehouse//RT warehouse

Lakehouse//RT has settings that control performance and cost:

• Query size controls the compute available to a single query.
• Autoscaling controls how much the warehouse scales out to serve concurrent queries.
• Auto stop controls how long an idle warehouse keeps running before it stops.

Query size and autoscaling work differently from the sizing and scaling settings on a serverless SQL warehouse.

Query size

The query size sets the maximum compute a single query can use, which determines how fast an individual query can run. It also sets the minimum compute the warehouse runs on, and the minimum you are billed for, while the warehouse is up.

You set the query size when you create the warehouse. Choose Small, Medium, Large, or X-Large. A larger size gives each query more compute for faster results, at a higher minimum cost.

This setting is similar in concept to the size of a serverless SQL warehouse, which also sets the compute available to a single query.

Autoscaling

Autoscaling lets a warehouse add compute to serve more concurrent queries, then release it when demand drops. You set the maximum compute the warehouse can scale up to, measured in DBUs.

Autoscaling on Lakehouse//RT differs from cluster scaling on a serverless SQL warehouse in a few ways:

• It is measured in DBUs, not clusters.
• The maximum is independent of query size. You can pair a small query size with a high maximum to get high concurrency without giving each query more compute.
• Lakehouse//RT autoscaling adds and removes just the compute needed, instead of in increments of clusters.

To see how much a warehouse is scaling, use its monitoring page.

Auto stop

Auto stop terminates the warehouse after it's idle for a set number of minutes, so an idle warehouse doesn't keep accruing charges. You set the idle timeout when you create the warehouse.

Auto stop on Lakehouse//RT works the same as on a serverless SQL warehouse. For the default and minimum values, see Configure SQL warehouse settings.

Monitor Lakehouse//RT activity

You can monitor Lakehouse//RT queries the same as any query run on a SQL warehouse:

• Query history: Lakehouse//RT queries appear in the query history UI and the query history system table.
• Query profiles: Open a Lakehouse//RT query in the query history UI to view its query profile.
• Monitoring page: Monitor query throughput, queued queries, and query history on the monitoring page for each Lakehouse//RT warehouse.
• Billing: Lakehouse//RT usage appears in the billing system tables with a sku_name of Lakehouse_Serverless.

Best practices

To get the best results from Lakehouse//RT, prepare your workloads before you move them over:

• Validate on serverless SQL first. Run your queries on a serverless SQL warehouse and confirm that they run in a few seconds.
• Use Unity Catalog managed tables. Managed tables with predictive optimization and liquid clustering ensure that your data is well clustered for your workload patterns.
• Verify queries are selective. For sub-second latency, verify that your queries scan lesser amounts of data. Filter early with WHERE clauses, select only the columns you require, and lean on aggregations. Joining across tables is supported, but if you find your query getting complex or slow, consider using materialized views that pre-aggregate your data for faster latencies.
• Check SQL coverage. Lakehouse//RT supports ANSI-compliant read queries only. Confirm that your workloads are ANSI compliant and avoid the unsupported statements, functions, and data types listed under Limitations.

Supported features

Tools and interfaces

You can select Lakehouse//RT from the compute picker in any of the following Azure Databricks features:

• SQL editor
• SQL notebooks
• AI/BI Dashboards
• Catalog Explorer
• Alerts

Table types

Lakehouse//RT queries Unity Catalog data only. For best performance, use Unity Catalog managed tables, which provide the engine with the data layout it needs for low latency.

Lakehouse//RT supports the following table types:

• Managed tables (Delta Lake and Apache Iceberg tables)
• Materialized views and streaming tables
• Metric views

Connectivity

Lakehouse//RT only accepts connections that use the Statement Execution API. It does not support the legacy Thrift protocol, so a driver that connects without explicitly using the Statement Execution API receives a 501 error.

You can connect to a Lakehouse//RT warehouse in the following ways:

• Statement Execution API: Call the API directly from external applications. See Statement Execution API: Run SQL on warehouses.
• Databricks drivers: The following drivers can connect when you configure them to use the Statement Execution API. Point the driver's HTTP path at your Lakehouse//RT warehouse, then set the following option:
  • Databricks SQL Connector for Python: Set use_kernel=True.
  • Databricks SQL Driver for Node.js: Set useKernel: true.
  • JDBC: Set UseThriftClient=0 in the connection URL.
  • ADBC driver for Power BI: Set Implementation=2.0 and Protocol=REST(SEA) under Advanced options.

Pricing

For pricing information, see the Lakehouse Real-Time pricing page.

Limitations

When a query uses an unsupported feature, Lakehouse//RT returns an error naming the feature. To successfully run the query, use a serverless SQL warehouse instead.

Tools and features

Lakehouse//RT does not yet support the following features:

• Genie
• Genie Agents
• Jobs tasks

Table types

The following table types are not yet supported:

• System tables
• Delta Sharing tables
• Tables in Unity Catalog default storage
• External tables in Unity Catalog

Lakehouse//RT does not support the following table types:

• Hive metastore tables (managed or external)
• Foreign tables and query federation (Lakehouse Federation)
• Temporary tables
• Tables that use other data formats (CSV, JSON, Avro, Parquet, ORC, and text)

Drivers and connectors

Lakehouse//RT does not support the following drivers and connectors:

• ADBC
• ODBC
• Go

SQL language

Lakehouse//RT runs SQL read queries in ANSI mode only. It evaluates all implicit type coercions and casts according to strict ANSI SQL rules, and this behavior cannot be turned off. Under ANSI semantics, queries that relied on non-ANSI behavior might:

• Raise a runtime error instead of silently producing NULL. For example, casting a non-numeric string to a number.
• Raise an analysis-time error when there is no safe common type. For example, COALESCE, CASE, IN, or set operations across incompatible types.
• Return a different result type than legacy mode, because ANSI string promotion and numeric widening choose safe, lossless types.

To get predictable results, use explicit CAST expressions when ANSI mode implicit conversions don't produce the behavior you expect.

Lakehouse//RT does not support the following:

• Data types: The GEOGRAPHY and GEOMETRY data types.
• Functions: AI functions, Python UDFs, spatial SQL functions, and XPath and XML functions.
• Governance: Attribute-based access control (ABAC), including row-level security and column masking.

Lakehouse//RT is for read (SELECT) queries only. Write and ETL commands are not supported, including:

• Write operations: INSERT, UPDATE, DELETE, MERGE, and CREATE TABLE AS SELECT (CTAS).
• DDL: CREATE, ALTER, DROP, and other statements that create or modify objects.
• Security statements: GRANT and REVOKE.
• Scripting, stored procedures, temporary tables, and multi-statement transactions.
• Delta Lake maintenance: OPTIMIZE, ANALYZE, VACUUM, and REFRESH.

Network security

Lakehouse//RT does not yet support the following network configurations:

• Outbound Private Link (serverless)

• Serverless egress control

Compliance

Compliance security profiles are not currently supported.