It is not every day you get to join a company working on building a product purpose-built for you.
It's not every day you get to join a company working on building a product purpose-built for you.
I still remember when I first heard about Dagster. Nick Schrock was on the Data Engineering Podcast: Build Maintainable And Testable Data Applications With Dagster in October 2019, discussing a new approach to orchestration. At the time, I was deep into Airflow, struggling with testing my pipelines before pushing them to prod, repeatedly trying to reason about schedules and start dates, and wondering why I had to install Kubernetes on my laptop.
So when I heard Nick talk about building a better way to connect data pipelines, I was sold. I still have a screenshot of my first message in the Dagster Slack. Back then, Dagster was a little rough around the edges, and some of the concepts from that time are long gone (solids, anyone?).
Even then, I kept my eye on the project as it evolved over the years, using it on the job and discovering firsthand that orchestration can be fun, too.
My career has had something to do with data for as long as I can remember, from working in various data roles at a bank, where I first learned how to automate data entry and calculations, to the fast-paced world of startups as a data scientist and, eventually, data engineer. In those 10+ years, it’s surprising that many of the challenges I started with persist to this day.
All data people want is to be able to transfer data, store it, transform it, visualize it, and operationalize it. We want the things we build to be maintainable, scalable, and observable.
There’s a saying that perfection is the enemy of good. I would add that good enough is also the enemy of good. As data people, we’ve been too complacent with satisfactory tooling, and the tide seems to be turning on that front. And one team that is unwavering in their pursuit of more than just good enough is the people here at Dagster Labs.
I’ve had the pleasure of working with Nick, Pete, and the rest of the team over the past year as I built integrations and projects with Dagster. I’ve never seen such a healthy mix of performance and compassion among so many people before. None of us want to build a better orchestrator; we want to rethink what’s possible with data engineering.
There's no shortage of excitement here, whether it’s software-defined assets, auto-materialization, or asset checks. Between a great product, a wonderful community, and a great team, I consider myself lucky to be part of Dagster Labs.
If you ever want to talk about your experience, have questions with Dagster, or even just to chat about data, don’t hesitate to reach out to me at pedram@dagsterlabs.com
