We are happy to announce that Python support for Databricks Asset Bundles is now available in Public Preview! Databricks users have long been able to author pipeline logic in Python. With this release, the full lifecycle of pipeline development—including orchestration and scheduling—can now be defined and deployed entirely in Python. Databricks Asset Bundles (or “bundles”) provide a structured, code-first approach to defining, versioning, and deploying pipelines across environments. Native Python support enhances flexibility, promotes reusability, and improves the development experience for teams that prefer Python or require dynamic configuration across multiple environments.
Standardize job and pipeline deployments at scale
Data engineering teams managing dozens or hundreds of pipelines often face challenges maintaining consistent deployment practices. Scaling operations introduces a need for version control, pre-production validation, and the elimination of repetitive configuration across projects. Traditionally, this workflow required maintaining large YAML files or performing manual updates through the Databricks UI.
Python improves this process by enabling programmatic configuration of jobs and pipelines. Instead of manually editing static YAML files, teams can define logic once in Python, such as setting default clusters, applying tags, or enforcing naming conventions, and dynamically apply it across multiple deployments. This reduces duplication, increases maintainability, and allows developers to integrate deployment definitions into existing Python-based workflows and CI/CD pipelines more naturally.
“The declarative setup and native Databricks integration make deployments simple and reliable. Mutators are a standout, they let us customize jobs programmatically, like auto-tagging or setting defaults. We’re excited to see DABs become the standard for deployment and more.”
— Tom Potash, Software Engineering Manager at DoubleVerify
Python-powered deployments for Databricks Asset Bundles
The addition of Python support for Databricks Asset Bundles streamlines the deployment process. Jobs and pipelines can now be fully defined, customized, and managed in Python. While CI/CD integration with Bundles has always been available, using Python simplifies authoring complex configurations, reduces duplication, and enables teams to standardize best practices programmatically across different environments.
Using the View as code feature in jobs you can also copy-paste directly into your project (Learn more here):
Advanced capabilities: Programmatic job generation and customization
As part of this release, we introduce the load_resources function, which is used to programmatically create jobs using metadata. The Databricks CLI calls this Python function during deployment to load additional jobs and pipelines (Learn more here).
Another useful capability is the mutator pattern, which enables you to validate pipeline configurations and update job definitions dynamically. With mutators, you can apply common settings such as default notifications or cluster configurations without repetitive YAML or Python definitions:
Learn more about mutators here.
Get started
Dive into Python support for Databricks Asset Bundles today! Explore the documentation for Databricks Asset Bundles as well as for Python support for Databricks Asset Bundles. We’re excited to see what you build with these powerful new features. We value your feedback, so please share your experiences and suggestions with us!
