Dagster & Modal
About this integration
Modal is a serverless cloud platform that makes it easy to run Python code at scale without managing infrastructure. The Dagster-Modal integration enables you to execute compute-intensive workloads, ML training jobs, and data processing tasks on Modal's auto-scaling platform directly from your Dagster pipelines.
With this integration, you can:
- Run serverless functions that automatically scale from zero to thousands of containers
- Access GPU resources for machine learning training and inference workloads
- Execute long-running jobs without worrying about infrastructure management
- Optimize costs with pay-per-use pricing and automatic scaling
- Deploy instantly with Modal's container orchestration
- Monitor execution through Dagster's observability tools
Modal is particularly powerful for ML workflows, data processing at scale, and any compute-intensive tasks that benefit from elastic infrastructure.
Installation
pip install dagster-modal modal
Key Features
Serverless Execution
Modal automatically provisions and scales compute resources based on demand, eliminating the need to manage servers or containers.
GPU Support
Access powerful GPU instances (A10G, A100, H100) for machine learning training and inference without long-term commitments.
Instant Deployment
Deploy functions and containers instantly with Modal's fast cold-start times and efficient resource allocation.
Cost Optimization
Pay only for actual compute time with automatic scaling to zero when idle, making it cost-effective for variable workloads.
Container Orchestration
Run complex applications with full control over the runtime environment, dependencies, and resource requirements.