May 20, 2025 • 3 minute read •

Beyond Point to Point

Why Modern Data Teams Need Orchestration, Not Just Integration

Name: Pedram Navid
Handle: @pdrmnvd

Data integration platforms like Workato have gained attention for their promise of codeless connectivity between applications. However, as many data engineers have discovered, these tools often fall short when faced with the complexities of modern data workflows. Let's examine why purpose-built data orchestration platforms provide a more robust foundation for scalable data engineering.

The Limitations of Traditional iPaaS Solutions

A recent Reddit discussion highlighted several pain points experienced by data engineers using integration platforms:

One user expressed frustration with the interface limitations:

I lose brain cells every time I work with it. How can anyone in their right mind build an interface people are supposed to work in and not include 'undo'?

Another pointed out the narrow use cases:

I can see it fit if you do some really simple stuff like 'if X happens here, then perform Y over in that other app', but even then, if it's not one of the 4 basic API calls they support out of the box, then you have to build a custom one anyway.

These experiences reveal a fundamental truth: while integration platforms can connect systems, they weren't designed with data engineers' workflows in mind.

Why Data Orchestration Platforms Offer a Better Approach

Modern data orchestration platforms address these limitations by providing:

Code-first, developer-friendly interfaces that integrate with existing engineering workflows
End-to-end observability across your entire data platform
Asset-centric architecture that focuses on data products rather than just connections

Example: Building Resilient Data Pipelines

Consider this typical integration scenario using a modern data orchestration approach:

@dg.asset
def customer_orders(context, raw_orders):
    """Process raw order data into a validated, transformed dataset"""
    # Apply transformations with proper error handling
    transformed_df = process_orders(raw_orders)

    # Built-in quality checks
    context.add_expectation_result(
        dg.ExpectationResult(
            success=not transformed_df.empty,
            label="orders_not_empty",
            metadata={"row_count": len(transformed_df)},
        )
    )

    return transformed_df

This approach provides several advantages:

Version control integration
Built-in testing capabilities
Clear lineage tracking
Automatic monitoring and alerting

Handling Complex Workflows

For organizations dealing with legacy systems, like the user who described their EOL ERP with 20 point-to-point interfaces, a robust orchestration platform offers significant advantages:

# Define a reusable connector that can be adapted for the new ERP
@dg.asset(
    io_manager_key="erp_connector",
    metadata={"system": "current_erp", "interface_type": "financial"},
)
def financial_transactions():
    # Logic to extract data from current ERP
    # Can be easily modified when migrating to new ERP
    # without changing downstream dependencies
    ...

With this architecture, the eventual migration becomes significantly easier as only the connector implementation needs to change, not the entire pipeline.

Building for Scale and Maintainability

While iPaaS solutions like Workato can solve immediate connection needs, data teams looking to build scalable, maintainable platforms should consider:

Developer experience: Tools should enhance productivity, not hinder it
Observability: Complete visibility into data flows and pipeline health
Reusability: Components that can be shared across the organization
Testing: Built-in capabilities for ensuring data quality

Conclusion

As one Reddit user aptly noted:

Coding was never the hard part of building things.

The real challenges lie in building reliable, maintainable data systems that can evolve with your organization's needs.

Modern data orchestration platforms address these fundamental challenges by providing a unified control plane for your data assets, enabling teams to build with confidence and scale without friction. Rather than focusing solely on point-to-point connections, these platforms help you create a cohesive data ecosystem that delivers trusted data to every stakeholder.

We're always happy to hear your feedback, so please reach out to us! If you have any questions, ask them in the Dagster community Slack (join here!) or start a Github discussion. If you run into any bugs, let us know with a Github issue. And if you're interested in working with us, check out our open roles!

Follow us: