Deep Dive: Building a Cross-Workspace Control Plane for Databricks

As Databricks deployments scale, a familiar pattern emerges: multiple workspaces, multiple teams, and no reliable way to manage the dependencies between them. Databricks Lakeflow Jobs is excellent for orchestrating work within a single workspace but it wasn't designed for the cross-workspace, cross-tool coordination that enterprise platforms require.

In this hands-on deep dive, we'll show you how to build a cross-workspace control plane using Dagster on top of your existing Databricks environment. We'll cover DatabricksWorkspaceComponent for auto-discovering workspace assets, Dagster Pipes for bidirectional notebook orchestration with live log streaming and metadata reporting, and the full reference architecture connecting integrations, and multiple Databricks workspaces into a single observable asset graph.

Demo-heavy and practitioner-focused, you'll leave with working patterns you can apply to your own platform the same day. We'll close with the incremental adoption path: how to start by observing your existing workspaces with zero code changes, then layer in orchestration and governance at your own pace.