Standalone pipelines vs. Lakeflow pipelines

Azure Databricks offers two ways to build materialized views and streaming tables: standalone pipelines, or Lakeflow pipelines. Both are built on Apache Spark™ Declarative Pipelines and produce Unity Catalog managed tables. The difference is how much of the pipeline you author and operate.

• A standalone materialized view or streaming table is a single dataset. Azure Databricks creates and manages a pipeline behind the scenes to refresh it. You create and refresh standalone datasets with SQL from a Databricks SQL warehouse, or from a notebook on serverless general compute with either spark.sql() or the Python pipelines decorators. See Standalone materialized views and streaming tables.
• A Lakeflow pipeline is a pipeline that you author and operate as a unit. It can contain many datasets, in SQL and Python, with dependency orchestration, lineage, and pipeline-wide operational features. See What are pipelines?.

When you create a standalone materialized view or streaming table, the managed pipeline appears on the Jobs & Pipelines page with a pipeline type of MV/ST. Datasets defined in a Lakeflow pipeline have a pipeline type of ETL.

When to use a standalone pipeline

Use standalone materialized views and streaming tables when:

• You accelerate queries or transform data with a single materialized view or streaming table.
• You work from a Databricks SQL warehouse, the SQL editor, or a notebook on serverless general compute, and schedule refreshes with SCHEDULE, TRIGGER ON UPDATE, or a SQL task in a job.
• You don't need sinks, multi-stage orchestration, or other pipeline-only features.

When to use a Lakeflow pipeline

Use a Lakeflow pipeline when:

• You build a multi-stage pipeline with intermediate datasets, where Azure Databricks manages dependencies and lineage across the datasets. Intermediate datasets can be published to the catalog or kept private to the pipeline.
• You author tables and flows in Python.
• You write to external Delta tables or event streaming destinations using sinks (create_sink() or foreach_batch_sink()).
• You apply change data capture from a database snapshot using create_auto_cdc_from_snapshot_flow().
• You want triggered or continuous execution across the whole pipeline.

Comparison