About this integration
Airlift is a migration toolkit that makes it easy to transition from Airflow to Dagster with minimal disruption to your existing workflows. Rather than requiring a complete rewrite, Airlift enables a gradual, task-by-task migration that can be completed in any order with minimal coordination between teams.
With Airlift, you can:
- Observe multiple Airflow instances from a centralized Dagster control plane
- Migrate individual tasks or entire DAGs incrementally
- Federate execution across multiple Airflow deployments
- Monitor both Airflow and Dagster assets in a unified interface
The toolkit is designed to work with your existing Airflow infrastructure, including AWS MWAA, Google Cloud Composer, and Astronomer deployments.
Installation
pip install 'dagster-airlift[core]' dagster-webserver dagster
Key Features
Incremental Migration
Migrate your Airflow DAGs to Dagster assets one task at a time, allowing teams to move at their own pace without breaking existing workflows.
Multi-Instance Observability
Connect multiple Airflow instances to a single Dagster deployment for centralized monitoring and orchestration across your entire data platform.
Asset-Centric Approach
Transform Airflow tasks into Dagster software-defined assets, providing better lineage tracking and dependency management.
Minimal Code Changes
Existing Airflow code requires minimal modifications during the migration process, reducing risk and development overhead.
Getting Started
Step 1: Peer Your Airflow Instance
Connect your Airflow instance to Dagster to begin observing DAG executions:
from dagster_airlift.core import (
AirflowBasicAuthBackend,
AirflowInstance,
build_defs_from_airflow_instance,
)
# Define your Airflow instance
airflow_instance = AirflowInstance(
auth_backend=AirflowBasicAuthBackend(
webserver_url="http://localhost:8080",
username="admin",
password="admin",
),
name="my_airflow_instance",
)
# Create Dagster definitions
defs = build_defs_from_airflow_instance(
airflow_instance=airflow_instance
)
Step 2: Observe Assets
Map your Airflow tasks to Dagster assets to track data lineage:
from dagster import AssetSpec
from dagster_airlift.core import assets_with_task_mappings
# Define asset specs for your data
customers_spec = AssetSpec(key="customers")
orders_spec = AssetSpec(key="orders")
# Map assets to Airflow tasks
mapped_assets = assets_with_task_mappings(
dag_id="etl_pipeline",
task_mappings={
"extract_customers": [customers_spec],
"extract_orders": [orders_spec],
}
)
Step 3: Migrate Tasks
Gradually migrate task execution from Airflow to Dagster:
# In your Airflow DAG file
from dagster_airlift.in_airflow import proxying_to_dagster
from dagster_airlift.in_airflow.proxied_state import load_proxied_state_from_yaml
# Enable proxying for specific tasks
if PROXYING_ENABLED:
proxying_to_dagster(
global_vars=globals(),
proxied_state=load_proxied_state_from_yaml(
Path(__file__).parent / "proxied_state.yaml"
),
)
Supported Airflow Versions
Airlift supports Airflow 2.0+ and requires access to Airflow's REST API:
- Airflow 2.0+: Full support with stable REST API
- AWS MWAA: Supported (Airflow 2.4.3 or later required)
- Google Cloud Composer: Full support
- Astronomer: Full support
- Self-hosted Airflow: Full support with proper API access
About Airlift
Airlift represents a paradigm shift in data platform migration, moving away from "big bang" rewrites toward incremental, low-risk transitions. By leveraging Airflow's REST API and Dagster's software-defined assets, Airlift provides a bridge between traditional workflow orchestration and modern data platform architecture.
The toolkit was designed based on real-world migration experiences, focusing on practical challenges like organizational coordination, system reliability, and business continuity during platform transitions.