Note
Access to this page requires authorization. You can try signing in or changing directories.
Access to this page requires authorization. You can try changing directories.
Zerobus Ingest is a push-based streaming API that writes data straight into Unity Catalog Delta tables at high scale, with no message bus to run. The workflow is two steps: create a table, then push data to it. Zerobus Ingest removes the need to manage partitions, brokers, or pipelines. It's a serverless endpoint that's on by default in your workspace and scales as you open more connections.
Built for high throughput and near real-time freshness, Zerobus Ingest handles high-volume concurrent writes from thousands of clients to the same table and lands records in Delta within seconds, so your data is query-ready almost as soon as it arrives.
- Zerobus Ingest is available in select regions. Your workspace and target table must both be in a supported region. For the list of supported regions, see Ingestion availability.
Built for high scalability
Zerobus Ingest is designed for high scalability, without you planning capacity. It has ingested over 1 trillion records into a single Delta table in under 24 hours, handling high-volume concurrent writes from thousands of clients, as described in the Ingesting the Milky Way: Petabyte-Scale with Zerobus Ingest blog post. For default throughput quotas, see Zerobus Ingest quotas.
A "hello world" client and a petabyte-scale workload run essentially the same code. You scale by running more producers, not by rewriting your application.
Zerobus Ingest is serverless: it adds and removes capacity as load changes. Streams act as dynamic partitioning units that the service opens, closes, and rotates to rebalance capacity as demand shifts.
To learn how Zerobus Ingest achieves this, see How Zerobus Ingest scales.
No message bus required
Many teams put a message bus such as Kafka between their producers and their tables purely to buffer data on its way into the lakehouse. That adds hops, cost, and operational overhead: brokers to size, partitions to rebalance, and consumer lag to watch. Zerobus Ingest removes that middle layer, allowing producers to write straight to Delta.
A message bus is still the right tool when the same data feeds many non-lakehouse consumers, when you need microservice-to-microservice messaging, or for message fanout. In those cases, when you also want that data in the lakehouse, use Azure Databricks managed streaming connectors to replicate from the message bus. But when the lakehouse is the destination, Zerobus Ingest is a simpler, more direct path.
How it works
A producer opens a stream to Zerobus Ingest and pushes records to a target Delta table. The service validates each record against the table schema and makes it durable. Once a record is durable, Zerobus Ingest acknowledges it quickly, so your producer can keep sending records without waiting on each one. The data is materialized into the table shortly afterward, typically within seconds. Zerobus Ingest's dynamic, partitionless design makes ingestion elastic, so its serverless compute scales with your workloads.
For a deeper explanation of streams and how Zerobus Ingest scales, see Zerobus Ingest concepts. For the asynchronous client and server communication model, see Asynchronous communication.
Create a table, then push data
Any application that can use a Zerobus Ingest SDK or call a supported API (gRPC, REST, or OpenTelemetry) can stream data into a Delta table. The table's schema defines what each record must contain. First, create the target table:
CREATE TABLE main.default.air_quality (
device_name STRING,
temp INT,
humidity INT
);
Then, after you grant a service principal access to the table, ingesting a record is a few lines of code:
from zerobus.sdk.sync import ZerobusSdk
from zerobus.sdk.shared import TableProperties
sdk = ZerobusSdk(SERVER_ENDPOINT, DATABRICKS_WORKSPACE_URL)
table_properties = TableProperties("main.default.air_quality")
stream = sdk.create_stream(CLIENT_ID, CLIENT_SECRET, table_properties)
stream.ingest_record_offset({"device_name": "sensor-1", "temp": 22, "humidity": 55})
stream.close()
The same code that ingests one record scales to petabytes: you run it from more producers. For the full walkthrough, see Use Zerobus Ingest.
When to use Zerobus Ingest
| Use Zerobus Ingest when… | Consider another tool when… |
|---|---|
| The lakehouse is the sole destination for your data. | You need to fan the same data out to many non-lakehouse consumers (use a message bus such as Kafka, and use Managed streaming connectors to replicate that data into the lakehouse). |
| You want high-throughput, concurrent writes directly to Delta tables. | You need microservice-to-microservice messaging (use a message bus). |
| Near real-time freshness (seconds) meets your needs. | You require sub-second operational latency in the processing path (use Real-time mode concepts). |
| You control the producer and can push data to an API. | You are pulling from files already landed in cloud storage (use Auto Loader). |
One design point to plan for: Zerobus Ingest guarantees ordering per stream, not globally across streams. For how per-stream ordering works and how to design around it, see Streams.
Common use cases
- IoT and device telemetry: stream sensor, vehicle, and smart-device data from large distributed fleets directly into governed Delta tables.
- On-premises to cloud: bridge on-prem and hybrid systems to the lakehouse without standing up broker infrastructure in between. For private connectivity and firewall setup, see Networking considerations.
- Application and clickstream events: push events from cloud and edge applications for near real-time analytics.
- Change data capture (CDC): land row changes from operational systems into Delta.
- Observability data: send OpenTelemetry traces, logs, and metrics into Delta tables you own. See Ingest OpenTelemetry data with Zerobus Ingest.
Ways to send data
Zerobus Ingest is one endpoint that supports multiple interfaces, so you can choose the best fit for each producer:
- SDKs over gRPC: high-throughput streaming clients in Python, Java, Rust, Go, TypeScript, and (in Beta) C++ and C# / .NET. Best for high-volume ordered ingestion. See Write a client.
- REST API: a stateless interface for lightweight or "chatty" clients such as large fleets of edge devices. See Write a client.
- OpenTelemetry (OTLP): point existing OpenTelemetry collectors at Zerobus Ingest to land traces, logs, and metrics with no custom integration. See Ingest OpenTelemetry data with Zerobus Ingest.
- Kafka-compatible APIs (Beta): point an existing Apache Kafka producer at Zerobus Ingest, with no Azure Databricks SDK. See Use Kafka-compatible APIs with Zerobus Ingest.
All of them write straight into Delta tables. For a full comparison and how to choose, see API protocols. To write your first client, see Use Zerobus Ingest.
Cost
Charges for Zerobus Ingest are billed against the "Automated Serverless" SKU. Pricing is available on the Lakeflow Connect pricing page.
Monitoring your usage
You can monitor your spend through the billable usage system table. See Billable usage system table reference. Filter for Zerobus Ingest usage with:
billing_origin_product = 'LAKEFLOW_CONNECT'product_features.lakeflow_connect.zerobus_request_typeidentifies how the data was ingested:'GRPC'(SDK streaming),'HTTP'(REST),'OTEL_GRPC'and'OTEL_HTTP'(OpenTelemetry/OTLP), or'KAFKA'(Kafka-compatible APIs).