3 posts tagged with "Data Integration"

Ask a data engineer whether their pipeline is ETL or ELT and you'll get an instant answer. Old-school engineers say ETL. dbt users say ELT.

Both answers are incomplete. There's a third pattern that more accurately describes what modern data pipelines actually do — and it's been hiding in plain sight: EtLT.

The Three Paradigms​

ETL: Transform Before You Land​

Raw data is extracted, passed through a dedicated transformation tier (Spark, DataStage, Informatica), and only then written to the destination warehouse.

Source ──► [Transform tier] ──► Destination

Pros: The warehouse always holds clean, business-ready data. Compliance controls are enforced before data ever lands.
Cons: The transform tier becomes a bottleneck. Schema changes require coordinated updates across multiple layers. Running a dedicated compute cluster for transforms is expensive.

ELT: Land First, Transform In-Place​

Popularized by dbt. Raw data lands directly in the warehouse (BigQuery, Snowflake, ClickHouse), and SQL does the transformations in place. The transform "tier" is just the warehouse itself.

Source ──► Destination (raw layer) ──► [SQL inside warehouse] ──► Business layer

Pros: Raw data is preserved for auditing. You reuse warehouse compute. Iteration is fast.
Cons: Sensitive fields — SSNs, email addresses, phone numbers — land in plaintext. There's no second chance to mask them once they're in the warehouse. Data quality issues only surface after Load, at which point downstream models may already be contaminated.

EtLT: A Lightweight Transform In-Transit​

Source ──► [tiny t] ──► Destination (sanitized raw layer) ──► [T inside warehouse]

The tiny t is a small set of row-level transformations that happen while data is in flight:

The big T (post-Load Transform) is where actual business logic lives: multi-table JOINs, metric calculations, aggregations, ML feature engineering.

Why Engineers Keep Overlooking EtLT​

The reason is tooling — not concept.

Legacy ETL tools (Informatica, DataStage) made transformations expensive and complex. Engineers overcorrected by pushing all logic into the pipeline, which made pipelines brittle.

Modern ELT tools (Airbyte, Fivetran) swung to the opposite extreme: move data from source to destination with almost no in-transit processing, then let dbt handle everything.

The gap neither camp fills: when you need both in-transit operations (masking, filtering) and complex analytical SQL in the destination, you end up with awkward workarounds in both tool families.

EtLT fills exactly that gap.

SeaTunnel Is Built for EtLT​

Apache SeaTunnel describes itself in its official documentation as an "EL(T) data integration platform" — the parentheses around T are intentional. The Transform step is lightweight and optional. This is not a marketing choice; it's an architectural constraint.

The Three-Layer Model Maps Directly to EtLT​

SeaTunnel's execution model has three stages: Source → Transform → Sink.

Source (E)
  └──► Transform (tiny t)  ← optional, lightweight processing
        └──► Sink (L)
              └──► [Warehouse SQL / dbt] (T)  ← big T lives here

SeaTunnel draws a clear line around what Transform can do. The official docs state:

"Transform can only be used for some simple transformations of data, such as converting a column to uppercase/lowercase, modifying column names, or splitting one column into multiple columns."

This describes the intended scope of the built-in transforms. In the current Zeta SQL Transform, JOIN and GROUP BY are not supported, so cross-table joins and aggregations belong in the destination system or another dedicated processing layer.

Built-in Transforms Cover the Typical tiny t Operations​

SeaTunnel provides both map and flat-map transform interfaces. Its current built-in transforms focus on processing individual records and schemas rather than cross-row aggregation, which makes them a practical fit for the tiny t layer.

Where EtLT Matters Most: CDC Pipelines​

Real-time CDC sync is one of SeaTunnel's core use cases — and it's also where pure ELT breaks down for compliance.

Here's the fundamental problem: once a value from a binlog event lands in your warehouse, there is no second chance to mask it. A phone number or SSN written to a ClickHouse table can't be un-written by a downstream dbt model. The original plaintext is already persisted.

EtLT solves what ELT cannot: it applies compliance transformations inside the only window that exists before the data reaches its destination.

A minimal configuration shape for a SeaTunnel CDC job with tiny t transforms is shown below. Replace the example credentials and endpoints before running it:

env {
  job.mode = "STREAMING"
}

source {
  MySQL-CDC {
    plugin_output = "raw_user_info"
    url = "jdbc:mysql://localhost:3306/orders"
    username = "seatunnel"
    password = "change-me"
    server-id = 5601-5604
    table-names = ["orders.user_info"]
  }
}

transform {
  # Mask phone numbers in-transit — the raw value never reaches the warehouse
  Replace {
    plugin_input = "raw_user_info"
    plugin_output = "masked_user_info"
    replace_field = "phone"
    pattern = "(\\d{3})\\d{4}(\\d{4})"
    replacement = "$1****$2"
    is_regex = true
  }
}

sink {
  # L: land into the ClickHouse raw layer
  Clickhouse {
    plugin_input = "masked_user_info"
    host = "clickhouse-host:8123"
    database = "raw"
    table = "user_info"
    username = "default"
    password = "change-me"
    primary_key = "id"
    support_upsert = true
    allow_experimental_lightweight_delete = true
  }
}

Once the data is in ClickHouse, you build wide tables, compute retention metrics, and run analytical queries — that's the big T. dbt models are a natural fit here.

SeaTunnel + dbt: Full-Stack EtLT​

dbt owns the big T after Load. SeaTunnel owns everything from source to Load (including tiny t). They're complementary, not competing.

Source
  └── SeaTunnel (E + tiny t + L)
        └── dbt (T)
              └── BI / ML

SeaTunnel handles data arrival and in-transit hygiene. dbt handles data modeling and business logic. Each tool does what it's good at and nothing more.

The Trap: Trying to Push Big T Into SeaTunnel​

Because SeaTunnel supports SQL Transform, engineers sometimes try writing GROUP BY aggregations there. The current Zeta SQL Transform explicitly rejects GROUP BY and JOIN queries.

If you need pre-Load aggregation, you have two real options:

1. Do it in the destination system — this is the whole point of EtLT and ELT.
2. Use a dedicated processing job outside the SeaTunnel transform chain — at that point you are building a full streaming ETL pipeline rather than keeping the transformation in the tiny t layer.

Clear boundaries make failures easier to trace. When tiny t and big T are collapsed into the same layer, it becomes nearly impossible to reason about where something went wrong.

Summary​

SeaTunnel is a natural fit for EtLT — not because it claims the label, but because its Source/Transform/Sink separation, current built-in transform scope, and explicit "EL(T)" positioning all point to the same shape: move data fast, sanitize lightly in-flight, and leave the heavy lifting to the destination.

Further Reading​

• Apache SeaTunnel — About — official EL(T) positioning (see the "SeaTunnel Work Flowchart" section)
• SeaTunnel Transform Connectors — full list of built-in lightweight transforms
• dbt — the natural partner for the big T in EtLT
• SeaTunnel API Deep Dive — deep dive into the single-row map contract

Tongcheng Travel's data channel evolved over several years into four parallel systems for offline transfer, real-time integration, Sqoop jobs, and SeaTunnel jobs. Each system solved a problem at a particular stage, but their overlapping capabilities, different execution engines, and separate operational models eventually became a barrier to platform-wide governance.

At an Apache SeaTunnel Meetup, Xiaochen Zhou, who works on the data platform at Tongcheng Travel, explained how the company consolidated those systems into a unified batch and streaming data channel based on the Apache SeaTunnel Zeta Engine. The project had three non-negotiable goals: keep the migration transparent to application teams, prove data consistency before switching traffic, and improve execution efficiency and operational stability.

This article summarizes the architecture, migration safeguards, AI-assisted task generation, validation design, and future direction presented in that session.

Over the past decade, the Modern Data Stack has changed how teams build data platforms. Data ingestion, lakehouses or cloud warehouses, dbt, orchestration, metrics layers, BI, and governance were split into modular building blocks that can be assembled more flexibly. That shift moved data platforms away from heavy, closed, tightly coupled systems and toward a more open and cloud-native engineering model.

But once teams run that model in production, another truth becomes clear: the quality, speed, and recoverability of the whole platform still depend heavily on the first step, data integration.

Because of that, the next key question in data engineering is no longer only "where data should live." It is also "how data enters the platform continuously, reliably, and in a governed way." From that perspective, Apache SeaTunnel is more than a connector catalog. It is better understood as a data integration runtime for modern platforms and future agent-driven systems.

1. Modern Data Stack made platforms more modular, but it still underestimates data ingress​

The Modern Data Stack deserves real credit. It separates responsibilities more clearly: ingestion handles extraction, a lakehouse or warehouse provides unified storage and compute, dbt handles much of the transformation layer, orchestration manages lifecycle, and governance, quality, metrics, and BI deliver value to the business.

That separation makes technology choices easier. But in real production systems, the ingestion layer is often the most underestimated part. "Just sync the data in" is rarely a connector-only problem. It quickly becomes an operational problem: CDC, incremental capture, schema drift, source pressure, recovery after failure, duplicate prevention, and downstream data quality.

In other words, Modern Data Stack helped teams appreciate lakehouses and modeling more deeply, but the next step is to recognize the engineering value of the integration layer itself. It decides whether every downstream layer sees complete, timely, consistent, and recoverable data.

2. From ETL to ELT to EtLT: the real shift is responsibility, not terminology​

Traditional ETL transforms data before loading it. That works well when rules are stable and business change is limited. The downside is a heavy front-loaded pipeline that becomes harder to maintain as change accelerates.

ELT moved more responsibility to the destination. Teams extract first, load into the lakehouse or warehouse, and transform later. That takes advantage of stronger storage and compute at the platform layer, preserves more raw data, and gives analytics teams more flexibility.

But pure ELT is not always the best answer. If everything is loaded first and cleaned up later, schema drift, dirty fields, CDC inconsistencies, sensitive columns, duplicate rows, and format differences all move into the platform unchanged. Lakehouses, dbt models, semantic layers, and analytics teams then pay the price for problems that should have been handled earlier.

That is why EtLT matters. EtLT does not mean going back to heavyweight ETL. It means adding a necessary lightweight transformation stage before loading: standardization, filtering, mapping, routing, CDC event normalization, and structure adaptation before data lands in the unified foundation. The big semantic transformation still belongs to dbt, SQL models, and semantic layers later.

3. SeaTunnel fits naturally in EtLT as a runtime, not just a list of connectors​

Once we look at EtLT, SeaTunnel's role becomes straightforward. It fits the E + lightweight t + L part of the stack: connecting heterogeneous sources, running CDC or incremental ingestion, applying lightweight transformation, and delivering data into lakehouses, engines, or downstream storage.

That is the difference between SeaTunnel and a simple connector catalog. The real question is not just whether a system can connect. It is whether it can keep running, recover from failure, handle state, and prepare data so downstream systems can consume it safely.

Seen that way, SeaTunnel is more accurately described as a data integration runtime for modern data platforms. Depending on the connector, engine, and configuration, it can support patterns such as checkpoint-based recovery, schema-aware handling, and reliable delivery. That is exactly the layer EtLT needs.

4. Why pure ELT starts to break down​

Many teams naturally start with pure ELT: if the lakehouse is powerful enough, load everything first and clean it up later. That works in early or exploratory stages. But once the environment includes frequent change, CDC, heterogeneous sources, and always-on production pipelines, the cracks show quickly.

Schema drift can land directly in the platform and make tables harder to manage. Dirty or sensitive columns can increase governance and compliance cost. CDC events from different systems can force downstream models to absorb unnecessary complexity. Duplicate or noisy records can make analytics depend on more and more compensating cleanup.

EtLT addresses this by improving the landing step. It is not about moving all business logic back upstream. It is about making sure the platform receives healthier input: standardized data, schema-aware handling, normalized CDC events, and the filtering or routing that should happen before storage.

5. In the agent era, SeaTunnel becomes more important as an execution layer​

If Modern Data Stack made tools modular, Agentic Data Stack makes tools callable. Future platforms will not only rely on people clicking through UIs, configuring jobs manually, and reading logs after the fact. They will increasingly include agents that interpret goals, reason over lineage and impact, choose actions, validate outcomes, and improve over time.

In that loop, agents handle understanding and decision-making. They identify intent, analyze dependencies and blast radius, decide whether a problem needs ETL, ELT, or EtLT, and choose the right tools and risk boundaries.

SeaTunnel's role is different. Through execution APIs, orchestration integrations, or higher-level control layers, it can serve as the data execution layer that turns plans into actual movement: synchronization, CDC ingestion, controlled reruns, selective reload workflows, and recoverable delivery. Agents are useful only when they can act through reliable systems, and that is where a runtime like SeaTunnel matters.

6. SeaTunnel is not just a component of Modern Data Stack, but part of the runtime foundation of Agentic Data Stack​

Viewed as a layered system, the next generation of data engineering platforms may look something like this: data sources at the bottom, then a data integration runtime, then a unified data foundation, then transformation and semantics, then orchestration and governance, and finally an agent layer on top.

In that structure, SeaTunnel sits in a critical position. It turns heterogeneous change from the real world into standardized, continuously ingestible input for the platform. Without that layer, the lakehouse is mostly a passive container. With it, the lakehouse has a better chance to become a reliable foundation of truth.

That is why SeaTunnel should not be treated as only a "sync tool." In the Modern Data Stack, it carries real production-grade integration responsibilities. In the Agentic Data Stack, it becomes part of the runtime infrastructure that allows data engineering systems to move from analysis to action.

Conclusion​

The shift from Modern Data Stack to Agentic Data Stack is not just about adding an LLM interface. It is about changing the operating model of data engineering: people define goals, systems generate actions, tools execute those actions, and feedback improves the next round.

In that world, lakehouses matter, dbt matters, orchestration matters, and governance matters. But the question of who brings real-world data into the platform in a reliable and recoverable way still points to the data integration runtime.

That is where Apache SeaTunnel stands out. It is not merely a catalog of connectors. It is one of the layers that makes the rest of the platform operational.