About this integration
Including SLURM in your Dagster pipelines brings modern developer experience to high-performance computing. With dagster-slurm, you can:
- Run the same Dagster assets on your laptop, CI pipeline, or Tier-0 supercomputers without code changes
- Get full observability of Slurm jobs directly in the Dagster UI with real-time logs and metrics
- Package environments automatically using Pixi for reproducible, portable execution
- Submit batch jobs through sbatch while maintaining Dagster's lineage tracking and retries
- Choose between Bash, Ray, or custom launchers for flexible execution
This community-maintained integration uses Dagster Pipes to connect Dagster's orchestration with SLURM's powerful scheduler, delivering the ergonomics of modern data platforms on HPC systems.
Installation
uv add dagster-slurm
Or with pip:
pip install dagster-slurm
Example
import dagster as dg
from dagster_slurm import ComputeResource, RayLauncher
@dg.asset
def training_job(context: dg.AssetExecutionContext, compute: ComputeResource):
completed = compute.run(
context=context,
payload_path="workloads/train.py",
launcher=RayLauncher(num_gpus_per_node=2),
resource_requirements={
"framework": "ray",
"cpus": 32,
"gpus": 2,
"memory_gb": 120,
},
extra_env={"EXP_NAME": context.run.run_id},
)
yield from completed.get_results()
Switch from ExecutionMode.LOCAL to ExecutionMode.SLURM and the same asset submits via sbatch, runs on the cluster, and streams logs back to your Dagster UI.
About SLURM
SLURM (Simple Linux Utility for Resource Management) is an open-source workload manager and job scheduler designed for Linux clusters. It's used by many of the world's supercomputing centers and research facilities to manage compute resources, handle job queues, and coordinate parallel processing across thousands of nodes.