2 posts tagged with "Architecture"

For the past two decades, most enterprise data engineering systems have been built on one default assumption:

People understand the system. The system executes the pipeline.

Engineers understand the business context, break a requirement into steps, write SQL, Spark jobs, shell scripts, synchronization tasks, and scheduling workflows, and then let the system run them. The scheduler does not need to understand the business. The sync engine does not need to understand the metric. It only needs to execute the predefined flow reliably.

That model supported the era of data warehouses, data lakes, BI reporting, and batch scheduling very well.

But now that assumption is starting to break down.

Enterprise data systems are becoming more complex in every direction:

• More data sources
• Longer pipelines
• Stronger real-time requirements
• Faster business changes
• More conflicting metric definitions
• More AI application data, model feedback data, vector indexes, and unstructured content

In this environment, enterprises do not just need more pipelines, and they do not just need a better Copilot that can write SQL faster.

They increasingly need a Data Engineering Agent that can understand the system, plan tasks, call tools, validate outcomes, and accumulate experience over time.

In that shift, Apache SeaTunnel becomes especially important.

Because in the agent era, it is not enough for a system to "think." It also has to connect to real data sources, capture changes, execute synchronization, process incremental updates, preserve consistency, and move data to target systems in a reliable and cost-effective way.

In other words:

The agent understands the goal and plans the action. SeaTunnel turns that action into real, reliable, and recoverable data movement.

That is why SeaTunnel is well positioned to become a core execution foundation in the evolution from ETL, ELT, and EtLT to agent-driven data engineering.

ETL to ELT: the first major shift​

Traditional ETL is straightforward:

1. Extract data from the source.
2. Transform it in an intermediate layer.
3. Load the processed result into the target system.

This model fit the early data warehouse era well.

At that time, data sources were relatively limited, the pipeline was easier to understand, and compute resources were more centralized. Enterprises wanted to clean the data, standardize the structure, and define the core logic before loading data into the warehouse.

At its core, ETL is a deterministic pipeline model.

Its key assumption is:

People define the process in advance. The system executes the process.

Later, with the rise of cloud warehouses, data lakes, lakehouse architectures, and elastic compute, ELT became more popular.

ELT changed the order:

1. Extract
2. Load
3. Transform inside the target platform

Instead of transforming everything before loading, enterprises started moving raw or near-raw data into a unified storage layer first, then using the target platform's compute power for downstream modeling and analytics.

ELT solved several ETL limitations:

• It reduced upfront processing complexity.
• It preserved more original data.
• It gave analysts and modeling teams more flexibility later.

But ELT also created a new problem.

If all transformation is delayed until after loading, then dirty source data, schema drift, type mismatches, CDC events, privacy fields, and format inconsistencies all arrive directly in the target system.

That might be acceptable in simple batch scenarios. It becomes much more expensive in real-time synchronization, CDC, multi-table sync, lakehouse ingestion, SaaS API ingestion, and AI-oriented data engineering.

That is where a third pattern becomes more useful:

EtLT

Why EtLT matters​

EtLT is not just a compromise between ETL and ELT.

A more useful way to understand it is:

Extract -> lightweight transform -> Load -> semantic Transform

That means:

• Extract the data
• Apply the minimum engineering transformations required to make the data usable
• Load it into a unified data foundation
• Apply business-level and semantic transformation later

The key idea is the distinction between lowercase t and uppercase T.

Lowercase t is not heavy business modeling. It is the engineering work that must happen before data enters the platform safely and consistently, such as:

• Field projection
• Type mapping
• Format normalization
• Primary key or partition field handling
• Sensitive field masking
• CDC event conversion
• Multi-table routing
• Schema evolution handling
• Pre-ingestion quality validation
• One-read, multi-write patterns
• Rate limiting and parallelism control

These transformations should not always be postponed to the target system. Otherwise, the lakehouse or warehouse becomes full of inconsistent, weakly governed, and semantically unclear raw data.

At the same time, lowercase t should not try to absorb all business logic.

Complex business definitions, KPI semantics, subject-area modeling, and cross-domain aggregation still belong to uppercase T, which should happen in the warehouse, lakehouse, semantic layer, or metric layer.

That is the value of EtLT:

Standardize the data engineering layer before loading, then apply business semantics after loading.

This is exactly the place where SeaTunnel fits naturally.

Its Source, Transform, and Sink architecture is well suited for the lowercase t in EtLT. It can connect heterogeneous systems, apply lightweight transformation during movement, handle CDC, adapt schemas, route multiple tables, and write the result into the target platform.

In an EtLT architecture, SeaTunnel is not just a data mover. It becomes the data integration runtime that prepares data before it enters the unified data foundation.

Why traditional ETL starts to struggle​

Traditional ETL is built for relatively stable pipelines.

You write the rules, draw the DAG, schedule the tasks, and fix failures when they happen.

But modern enterprise data environments are no longer that simple.

Today a single enterprise may operate across:

• OLTP databases
• Kafka streams
• CDC pipelines
• SaaS APIs
• Object storage
• Logs and events
• Lakehouse platforms
• Real-time OLAP systems
• Vector databases
• AI interaction logs
• Model output datasets

The problem is not only that there is more data. The data is also more fragmented, more heterogeneous, and more real-time.

Pipeline length is another issue.

A single business metric may depend on dozens of tables, multiple layers of wide tables, several business domains, and a long chain of definition changes. At that point, many enterprises no longer struggle with "Can we build the workflow?" They struggle with "Can anyone still explain the whole pipeline end to end?"

This is where traditional ETL shows a structural limitation.

• One renamed field can break hundreds of tasks.
• One changed enum can silently shift multiple core metrics.
• One incorrect incremental logic branch can pollute an entire downstream analysis chain.

The scheduler can tell you that a task failed. It usually cannot tell you why that task matters.

The sync tool can move the data. It usually cannot tell you which business metric is now at risk.

The engineer can fix the script. But only if that engineer can first rebuild the missing context.

So the real weakness of traditional ETL is not just performance or reliability.

It is that:

It can execute the process, but it does not understand the system.

Why Copilot is not enough​

Many teams first bring AI into data engineering through Copilot-style workflows:

• Generate SQL
• Complete Spark code
• Draft YAML
• Produce test samples

These capabilities are useful. They improve local productivity.

But they do not solve the deepest problem in enterprise data engineering.

Because the hardest part of data engineering is rarely just code generation.

It is system understanding.

Copilot can help generate a SQL statement, but it does not know the real business meaning of the field.

It can help draft a synchronization task, but it does not know which downstream metrics will be affected by a schema change.

It can help generate a scheduler config, but it does not know whether the change breaks historical consistency or recovery semantics.

What enterprises actually struggle with includes:

• Lineage reasoning
• Dependency analysis
• Semantic understanding
• Metric governance
• Risk estimation
• Impact analysis
• Incremental recovery

These are not just autocomplete problems.

So enterprises do not only need an AI IDE. They increasingly need an agentic data engineering system that can understand the target, decompose tasks, call engineering tools, and verify the result.

The real shift: from pipeline to agent​

If we keep only one conclusion, it is this:

Traditional ETL is "people define the process, systems execute the process." Agentic data engineering is "people define the goal, systems generate the process."

That is not a slogan. It is a change in how work is organized.

In the traditional model, engineers design the task chain first, configure Source, Transform, and Sink, and then let the scheduler execute the pipeline.

The system faces a fixed process.

In the agent model, the input may only be a business goal.

For example:

Add a new gross margin metric for orders and keep it aligned with the finance definition.

Traditionally, the engineer must:

• Identify relevant data sources
• Read table schemas
• Inspect lineage
• Design transformation logic
• Configure sync and scheduling jobs
• Add quality checks
• Run regression validation

In an agent-oriented workflow, the system should be able to generate a sequence of actions around the goal:

• Identify the affected business entities
• Discover candidate data sources
• Analyze upstream lineage
• Decide whether the job belongs to ETL, ELT, or EtLT
• Generate or update the SeaTunnel synchronization task
• Configure full-load or CDC mode
• Apply lightweight transformation
• Write the result into the warehouse or lakehouse
• Trigger data quality validation
• Evaluate downstream impact
• Present the result for human confirmation

That is the real difference.

The breakthrough is not "AI wrote a SQL statement for me."

The breakthrough is:

The system starts generating engineering actions from a business goal.

But this immediately raises a critical question:

When the agent plans a data action, who executes it reliably?

That is exactly where SeaTunnel becomes essential.

SeaTunnel in the agent era: the data integration execution layer​

An agent cannot stop at reasoning and recommendations.

If a Data Engineering Agent decides that a table should be synchronized, a CDC job should be adjusted, a broken pipeline segment should be replayed, or a data slice should be reloaded into the target system, it needs a stable and observable execution layer to carry out that decision.

That execution layer needs several core capabilities.

1. It must connect to many kinds of data sources​

Enterprise data systems are inherently heterogeneous.

An agent cannot live in a world with only one database or one file system. It needs to connect to MySQL, Oracle, PostgreSQL, SQL Server, Kafka, Hive, Iceberg, Doris, ClickHouse, StarRocks, Elasticsearch, S3, HDFS, MongoDB, and many other systems.

SeaTunnel's connector architecture is designed for exactly this kind of environment. It abstracts Source, Transform, and Sink through a consistent plugin model so heterogeneous systems can be integrated in a unified way.

2. It must support batch, streaming, CDC, and large-scale synchronization​

The agent era does not run on a single data movement pattern.

It needs:

• One-time full migration
• Continuous CDC
• Offline batch movement
• Real-time synchronization
• Single-table sync
• Multi-table or database-level sync

SeaTunnel is valuable here because it is not just a script wrapper for ETL. It is a real data integration runtime that can support full load, incremental sync, real-time processing, CDC, and multi-table movement in the same ecosystem.

3. It must handle the lowercase t in EtLT​

Agentic systems do not need every business transformation to happen inside the sync layer.

But they do need the sync layer to complete the minimum engineering transformation required to make the data trustworthy and usable before it lands in the platform.

SeaTunnel's Transform layer is a strong fit for:

• Field mapping
• Type conversion
• Filtering
• Column projection
• Data masking
• Routing
• Simple reshaping

That is exactly the role of the lowercase t in EtLT:

Do not overload the movement layer with heavy business modeling, but make the data governable and ready for the next stage.

4. It must provide consistency, fault tolerance, and recovery​

An agent can decide that a broken link should be replayed.

But replay only matters if the underlying system can recover correctly.

The execution layer still needs checkpointing, failure recovery, state handling, restart behavior, and strong delivery guarantees where needed.

A reasoning layer without a reliable execution layer becomes a planner without hands.

That is why execution quality still matters as much as intelligence.

What the future stack starts to look like​

If we look one step ahead, enterprise data engineering increasingly resembles a layered operating system rather than a collection of disconnected pipelines.

In that stack:

• The semantic layer defines the business model.
• Metadata provides structure and context.
• Memory accumulates operational experience.
• The planning layer turns goals into actions.
• The execution layer performs synchronization, CDC, movement, replay, and recovery.

SeaTunnel belongs to this execution layer.

That placement is important.

The future is not "put a large language model on top of ETL."

The future is a coordinated system where reasoning and execution are separated clearly:

• The agent decides what should happen.
• SeaTunnel ensures that it actually happens in a reliable way.

The evolution in one sentence​

ETL built data pipelines.

ELT moved raw data into a unified platform first.

EtLT rebalanced pre-load engineering standardization and post-load semantic modeling.

The agent era pushes data engineering one step further:

From fixed pipelines to goal-driven systems.

In that world, SeaTunnel is not just a synchronization tool.

It becomes a practical execution foundation for agentic data engineering.

Agents make data systems understand goals.

EtLT makes ingestion more controllable.

SeaTunnel turns those goals into reliable data engineering actions.

That is the deeper change now happening across enterprise data engineering.

In the field of data integration and synchronization, Apache SeaTunnel is undoubtedly one of the most popular tools today. This series will dive deep into its advanced usage.

The first article starts with one of SeaTunnel’s core concepts: Data Flow. It analyzes the underlying principles, such as how data flows and is transformed, and explains how this concept applies to complex scenarios through examples.

One-Sentence Summary: The Conclusion First​

SeaTunnel is not a linear “source → sink” tool.

• It is a DAG execution engine driven by DataStream / DataFlow.

In SeaTunnel's Zeta engine, multiple upstream branches converging on one downstream stage is a direct manifestation of this model.

1. SeaTunnel’s Core Concept: Data Flow​

Inside SeaTunnel, everything revolves around data flow.

What is a data flow?​

Data flow = a stream of Records with a consistent structure and Schema.

It is not a table, not a file, and not a SQL result set.

It is:

Record1 → Record2 → Record3 → ...

Every plugin “operates on data flow”​

In practice, a plugin either produces a stream, consumes a stream, or does both when it sits in the middle of a pipeline.

2. The Real Meaning of plugin_output / plugin_input​

You may have been “using” them for a long time, but now it is time to truly “understand” them.

1️⃣ plugin_output​

plugin_output = "source_data_output_1"

Its meaning is not simply “a name”.

It means:

Assign a unique ID to the data flow produced by the current plugin.

You can understand it as:

DataStream<ID = source_data_output_1>

2️⃣ plugin_input​

plugin_input = "source_data_output_1"

Its meaning is:

This plugin wants to consume a specific data flow.

In one sentence​

plugin_output / plugin_input = the “connection ports” of data flow

3. SeaTunnel’s DAG Model: You Are Already Using It​

Under SeaTunnel's Zeta engine and its LogicalDag model, the successful experiment you ran is essentially this:

SourceA ┐
        ├──► Downstream Stage
SourceB ┘

Internally, SeaTunnel builds a DAG like this:​

DataStream A ┐
             ├──► Downstream Vertex
DataStream B ┘

Key question: Why can they converge?​

Because:

In Zeta's logical graph, one downstream vertex can be connected to multiple upstream data flows.

That is a graph-level capability first, not a blanket guarantee that every runtime plugin instance accepts multiple plugin_input values directly.

In Zeta, SeaTunnel internally does the following:

• Takes multiple input streams;
• Represents them as a multiple-input vertex in the LogicalDag;
• Splits the graph into executable pipelines during physical planning.

A precise caveat is important here:

• Zeta's DAG model can express multiple-input vertices.
• The current Flink and Spark starter implementations still reject multiple plugin_input values in a single plugin instance.
• So treat this as a Zeta / logical-graph concept, not as a universal sink-plugin contract.

4. How Is This Fundamentally Different from “SQL / ETL” Thinking?​

This is where many people get confused.

The SQL world​

SELECT * FROM A
UNION ALL
SELECT * FROM B

• This is result-set semantics.

The SeaTunnel world​

Record stream from A
Record stream from B
↓
A downstream stage processes the records according to the configured graph

• This is stream semantics.

In Zeta, as long as the graph and Schema assumptions are valid, the downstream stage can be planned from those upstream streams.

5. The Role of Schema in Data Flow: Must Remember​

Data flow = Record + Schema

Prerequisites for data flow convergence in SeaTunnel:​

• The number of fields must be consistent.
• Field types must be compatible.
• Field names must be aligned, or at least mappable.

Otherwise:

• The job may fail at runtime.
• Or the Sink may fail to write the data.

When you said “the target fields are definitely aligned”, that is exactly why your experiment succeeded.

6. Formal Definition of SeaTunnel’s Data Flow Model​

You can directly use the following standard wording in future design discussions, solution explanations, or documentation:

SeaTunnel uses DataStream as its core abstraction.

Source plugins generate data streams, Transform plugins process data streams and output new data streams, and Sink plugins consume upstream data streams and write the data into external systems.

In SeaTunnel Zeta's LogicalDag, multiple data streams can converge at one downstream vertex. Whether that convergence is expressed directly by a specific sink plugin or through an intermediate stage depends on the engine/runtime implementation.

7. Direct Impact on Your Builder / Strategy Design​

Now you can be very certain about three things:

1️⃣ The Builder must support N Sources → M Sinks​

It is not a 1→1 model. It is a graph model.

2️⃣ plugin_output is a “first-class citizen”​

If someone does not set plugin_output in your Builder:

• You should automatically generate one for them.

This is a platform-level capability.

3️⃣ In Zeta, the logical graph may contain multiple-input vertices​

Even if the DSL shows only one:

plugin_input = "s1"

Your Builder should model upstream relationships as:

Set<DataStream>

instead of hard-coding every downstream step as a single linear String-to-String hop.

8. Key Facts You Have Already Verified​

Here are the conclusions you have already validated through practice:

✅ SeaTunnel is a DAG, not a linear ETL tool.
✅ In Zeta's DAG model, multiple upstream branches can converge on one downstream stage.
✅ That convergence is a graph-level concept, not automatically a direct multi-input sink contract in every engine implementation.
✅ Schema alignment is the prerequisite.
✅ The DSL describes data flow, not SQL.

9. Summary​

SeaTunnel has only three core roles​

Source     →   Transform   →   Sink
(produces)     (modifies)      (consumes)
data flow      data flow       data flow

How are data flows “connected”?​

Just remember this connection rule:

The connection depends on two things:

• plugin_output: What is the name of the data flow I produce?
• plugin_input: Which upstream data flow am I consuming?

For example, two upstream branches converging on one downstream stage in Zeta:

SourceA ┐
        ├──► Downstream Stage
SourceB ┘

One Source → two Sinks:

         ┌──► SinkA
Source ──┤
         └──► SinkB

Two independent flows in one conf file:

SourceA ───► SinkA

SourceB ───► SinkB