Web3j

Web3j source connector

Support Those Engines

Spark
Flink
SeaTunnel Zeta

Key Features

• batch
• stream
• exactly-once
• column projection
• parallelism
• support user-defined split

Description

Source connector for Web3j. It reads blockchain data through a Web3 provider endpoint. Currently, the connector reads the latest block number and emits one value field. The value field is a JSON string that contains blockNumber and the read timestamp.

In batch mode, the source emits one row and then finishes. In streaming mode, it keeps polling the provider and emits the latest observed block number.

Source Options

Name	Type	Required	Default	Description
url	String	Yes	-	Web3 provider endpoint used to communicate with the Ethereum network, for example an Infura URL.

Output Schema

Field	Type	Description
value	String	JSON string containing the latest block number and a timestamp generated by the connector.

The JSON stored in value has this shape:

{"blockNumber":19525949,"timestamp":"2024-03-27T13:28:45.605Z"}

Example

env {
  parallelism = 1
  job.mode = "BATCH"
}

source {
  Web3j {
    url = "https://mainnet.infura.io/v3/xxxxx"
    plugin_output = "web3j"
  }
}

sink {
  Console {
    plugin_input = "web3j"
    parallelism = 1
  }
}

Then you will get data similar to the following:

{"value":"{\"blockNumber\":19525949,\"timestamp\":\"2024-03-27T13:28:45.605Z\"}"}

Changelog

Change Log

Change	Commit	Version
[improve] update Web3j connector config option (#9005)	https://github.com/apache/seatunnel/commit/9204f289d8	2.3.10
[Feature][Connector-V2] Add web3j source connector (#6598)	https://github.com/apache/seatunnel/commit/b7002bfaf4	2.3.6