About this integration
Dagster comes with a native PipesSubprocessClient resource that enables you to launch shell commands directly from Dagster assets and ops. This integration allows you to pass parameters to external shell scripts while Dagster receives real-time events, such as logs, asset checks, and asset materializations, from the initiated external execution. With minimal code changes required on the job side, this integration is both efficient and easy to implement.
Installation
pip install dagster
Example
import shutil
import dagster as dg
@dg.asset
def shell_asset(
context: dg.AssetExecutionContext, pipes_subprocess_client: dg.PipesSubprocessClient
) -> None:
shell_script_path = "/path/to/your/script.sh"
return pipes_subprocess_client.run(
command=["bash", shell_script_path],
context=context,
).get_results()
defs = dg.Definitions(
assets=[shell_asset],
resources={"pipes_subprocess_client": dg.PipesSubprocessClient()},
)
About Shell
A shell is a computer program that presents a command line interface which allows you to control your computer using commands entered with a keyboard instead of controlling graphical user interfaces (GUIs) with a mouse/keyboard/touchscreen combination.