Muokkaa

DataOps for the modern data warehouse

Azure Data Factory
Azure Databricks
Azure DevOps
Azure Key Vault
Azure Synapse Analytics

This article describes how a fictional city planning office could use this solution. The solution provides an end-to-end data pipeline that follows the MDW architectural pattern, along with corresponding DevOps and DataOps processes, to assess parking use and make more informed business decisions.

Architecture

The following diagram shows the overall architecture of the solution.

Architecture diagram demonstrating DataOps for the modern data warehouse.

Download a Visio file of this architecture.

Data flow

Azure Data Factory orchestrates and Azure Data Lake Storage Gen2 stores the data.

The following data flow corresponds to the previous diagram:

  1. The Contoso city parking web service API is available to transfer data from the parking spots.

  2. There's a data factory copy job that transfers the data into the Landing schema.

  3. Next, Azure Databricks cleanses and standardizes the data. It takes the raw data and conditions it so data scientists can use it.

  4. If validation reveals any bad data, it gets dumped into the Malformed schema.

    Important

    People have asked why the data isn't validated before it's stored in Data Lake Storage. The reason is that the validation might introduce a bug that could corrupt the dataset. If you introduce a bug at this step, you can fix the bug and replay your pipeline. If you dumped the bad data before you added it to Data Lake Storage, then the corrupted data is useless because you can't replay your pipeline.

  5. There's a second Azure Databricks transform step that converts the data into a format that you can store in the data warehouse.

  6. Finally, the pipeline serves the data in two different ways:

    1. Databricks makes the data available to the data scientist so they can train models.

    2. Polybase moves the data from the data lake to Azure Synapse Analytics and Power BI accesses the data and presents it to the business user.

Components

  • Azure Data Factory is a cloud-based data integration service that enables data movement and orchestration. In this architecture, it initiates the pipeline by copying data from the Contoso city parking web service API into the landing zone of the data lake.

  • Azure Data Lake Storage Gen2 is a scalable and secure data lake built on Azure Blob Storage that supports tiered storage and replayable pipelines. In this architecture, it serves as the central repository for both raw and processed data across landing, malformed, and validated data zones.

  • Azure Databricks is an Apache Spark-based analytics platform designed for big data and machine learning. In this architecture, it performs two critical transformation steps. First, it cleanses and standardizes raw data while filtering malformed records to a separate schema. Then it converts validated data into a format suitable for data warehouse storage and makes processed data available to data scientists for model training.

  • Azure Key Vault is a secure cloud service for managing secrets, keys, and certificates. In this architecture, it stores sensitive configuration settings and credentials used throughout the pipeline, providing centralized and secure configuration management.

  • Azure Synapse Analytics is an integrated analytics service that combines big data and data warehousing capabilities. In this architecture, it serves as the data warehouse that ingests transformed data from Data Lake Storage via PolyBase for querying and reporting.

  • Power BI is a business analytics tool that delivers interactive visualizations and dashboards. In this architecture, it connects to Azure Synapse Analytics to present parking usage data insights to city planners for informed decision-making.

Scenario details

A modern data warehouse (MDW) lets you easily bring all of your data together at any scale. It doesn't matter if it's structured, unstructured, or semi-structured data. You can gain insights to an MDW through analytical dashboards, operational reports, or advanced analytics for all your users.

Setting up an MDW environment for both development (dev) and production (prod) environments is complex. Automating the process is key. It helps increase productivity while minimizing the risk of errors.

This article describes how a fictional city planning office could use this solution. The solution provides an end-to-end data pipeline that follows the MDW architectural pattern, along with corresponding DevOps and DataOps processes, to assess parking use and make more informed business decisions.

Solution requirements

  • Ability to collect data from different sources or systems.

  • Infrastructure as code: deploy new dev and staging (stg) environments in an automated manner.

  • Deploy application changes across different environments in an automated manner:

    • Implement continuous integration and continuous delivery (CI/CD) pipelines.

    • Use deployment gates for manual approvals.

  • Pipeline as Code: ensure the CI/CD pipeline definitions are in source control.

  • Carry out integration tests on changes using a sample data set.

  • Run pipelines on a scheduled basis.

  • Support future agile development, including the addition of data science workloads.

  • Support for both row-level and object-level security:

    • The security feature is available in SQL Database.

    • You can also find it in Azure Synapse Analytics, Azure Analysis Services and Power BI.

  • Support for 10 concurrent dashboard users and 20 concurrent power users.

  • The data pipeline should carry out data validation and filter out malformed records to a specified store.

  • Support monitoring.

Potential use cases

This article uses the fictional city of Contoso to describe the use case scenario. In the narrative, Contoso owns and manages parking sensors for the city. It also owns the APIs that connect to and get data from the sensors. They need a platform that will collect data from many different sources. The data then must be validated, cleansed, and transformed to a known schema. Contoso city planners can then explore and assess report data on parking use with data visualization tools, like Power BI, to determine whether they need more parking or related resources.

Street Parking Availability

Considerations

These considerations implement the pillars of the Azure Well-Architected Framework, which is a set of guiding tenets that can be used to improve the quality of a workload. For more information, see Microsoft Azure Well-Architected Framework.

The considerations in this section summarize key learnings and best practices demonstrated by this solution:

  • Use data tiering in your data lake. Retain unmodified source data in the landing zone, route records that fail validation to the malformed zone, and transform validated data into a warehouse-ready format. Retaining source data lets you reprocess it without returning to the source system. For the analogous bronze, silver, and gold lakehouse pattern, see medallion architecture.

  • Make your data pipelines replayable and idempotent. Design transformation steps so that rerunning them over the same input produces the same result. Replaying a pipeline lets you fix a defect in transformation logic and reprocess historical data instead of discarding it.

Security

Security provides assurances against deliberate attacks and the abuse of your valuable data and systems. For more information, see Design review checklist for Security.

  • Secure and centralize configuration. Store connection strings, keys, and other secrets in Key Vault instead of in notebooks, pipeline definitions, or source control. Reference them from Data Factory linked services and from Azure Databricks secret scopes so that each environment resolves its own values.

Operational Excellence

Operational excellence covers the operations processes that deploy an application and keep it running in production. For more information, see Design review checklist for Operational Excellence.

  • Validate data early in your pipeline. Apply schema and quality checks in the first transformation step and route records that fail into the malformed schema. Early validation keeps defective records out of downstream tiers and gives you a record of what was rejected and why.

  • Ensure data transformation code is testable. Factor transformation logic into functions and modules that run outside a notebook so that you can cover them with unit tests in the pull request validation pipeline.

  • Have a CI/CD pipeline. Build and release every environment from source control rather than by hand. For the technology-specific mechanics, see CI/CD in Azure Data Factory and CI/CD on Azure Databricks.

  • Monitor infrastructure, pipelines, and data. Collect metrics and logs from each layer so that a failure surfaces as an alert instead of as a stale report. For more information, see Monitor Data Factory.

Deploy this scenario

The following list contains the high-level steps required to set up this solution with corresponding build and release pipelines.

Setup and deployment

  1. Initial setup: Install any prerequisites, create the Git repository that holds the infrastructure, notebook, and pipeline code, and set required environment variables.

  2. Deploy Azure resources: Use an infrastructure as code deployment, such as Bicep or Terraform, to deploy the Azure resources and Microsoft Entra service principals for each environment. Separately configure the Azure Pipelines definitions, variable groups, and service connections that invoke the infrastructure deployment.

  3. Set up Git integration in dev Data Factory: Configure Git integration so that the development data factory commits to your repository.

  4. Carry out an initial build and release: Create a sample change in Data Factory, like enabling a schedule trigger, then watch the change automatically deploy across environments.

Continuous integration and continuous delivery (CI/CD)

The following diagram demonstrates the CI/CD process and sequence for the build and release pipelines.

Diagram that shows the process and sequence for build and release.

Download a Visio file of this architecture.

  1. Developers develop in their own sandbox environments within the dev resource group and commit changes into their own short-lived Git branches. For example, <developer_name>/<branch_name>.

  2. When changes are complete, developers raise a pull request (PR) to the main branch for review. Doing so automatically kicks off the PR validation pipeline, which runs the unit tests, linting, and data-tier application package (DACPAC) builds.

  3. On completion of the PR validation, the commit to main will trigger a build pipeline that publishes all necessary build artifacts.

  4. The completion of a successful build pipeline will trigger the first stage of the release pipeline. Doing so deploys the publish build artifacts into the dev environment, except for Data Factory.

    Developers manually publish to the dev Data Factory from the collaboration branch (main). The manual publishing updates the Azure Resource Manager templates in the adf_publish branch.

  5. The successful completion of the first stage triggers a manual approval gate.

    On Approval, the release pipeline continues with the second stage, deploying changes to the stg environment.

  6. Run integration tests to test changes in the stg environment.

  7. Upon successful completion of the second stage, the pipeline triggers a second manual approval gate.

    On Approval, the release pipeline continues with the third stage, deploying changes to the prod environment.

For more information about implementing these stages, see CI/CD in Azure Data Factory.

Testing

The solution includes support for both unit testing and integration testing. Unit tests cover the Python transformation modules, and integration tests trigger a Data Factory pipeline and verify its output as part of the release to the staging environment. For more information, see Unit testing for notebooks.

Observability and monitoring

The solution supports observability and monitoring for Databricks and Data Factory. For Databricks, use the platform's built-in audit logging (the system.access.audit table) and job run monitoring rather than exporting all diagnostics by default. If you need to deliver Databricks diagnostic logs to a Log Analytics workspace for centralized alerting, that capability requires the Premium plan, applies to logs rather than metrics, and needs careful access control because audit logs can contain sensitive details about your deployment. For Data Factory, route diagnostic logs and metrics to a Log Analytics workspace and set up alerts on pipeline failures and job latency. For more information, see Monitor Data Factory.

Next steps

The following resources help you implement the DataOps practices that this article describes.

Continuous integration and delivery

Observability/monitoring

Azure Databricks

Data Factory

Azure Synapse Analytics

Azure Storage

Resiliency and disaster recovery

Azure Databricks

Data Factory

Azure Synapse Analytics

Azure Storage

Detailed overview

For a detailed overview of the solution and key concepts, watch the following video recording: DataDevOps for the Modern Data Warehouse on Microsoft Azure