HugeGraph Sink Connector

Sink: HugeGraph

Description

The HugeGraph sink connector allows you to write data from SeaTunnel to Apache HugeGraph, a fast and scalable graph database.

This connector supports writing data as vertices or edges, providing flexible mapping from relational data models to graph structures. It is designed for high-performance data loading.

Key Features

• batch
• exactly-once
• cdc
• support multiple table write
• timer flush

The connector writes rows as either vertices or edges. It supports insert, update, and delete row kinds, and it can flush buffered records by batch_size or batch_interval_ms.

caution

The HugeGraph schema must already exist before the job writes data. Create the required property keys, vertex labels, and edge labels in HugeGraph first, then use schema_config to map SeaTunnel fields to that existing graph schema.

Configuration Options

Name	Type	Required	Default Value	Description
host	String	Yes	-	The host of the HugeGraph server.
port	Integer	Yes	-	The port of the HugeGraph server.
graph_name	String	Yes	-	The name of the graph to write to.
graph_space	String	No	-	The graph space of the graph to be operated on.
username	String	No	-	The username for HugeGraph authentication.
password	String	No	-	The password for HugeGraph authentication.
batch_size	Integer	No	500	The number of records to buffer before writing to HugeGraph in a single batch.
batch_interval_ms	Integer	No	5000	The maximum time in milliseconds to wait before flushing a batch.
max_retries	Integer	No	3	The maximum number of times to retry a failed write operation.
retry_backoff_ms	Integer	No	5000	The backoff time between retries in milliseconds.

Sink Options

Name	Type	Required	Default Value	Description
schema_config	Object	Yes	-	The configuration for mapping the input data to HugeGraph's schema (vertices or edges).
selected_fields	List	No	-	A list of fields to be selected from the input data. If not specified, all fields will be used.
ignored_fields	List	No	-	A list of fields to be ignored from the input data. Mutually exclusive with selected_fields.

selected_fields and ignored_fields are applied before the row is mapped to HugeGraph. Keep every field used by idFields, sourceConfig.idFields, targetConfig.idFields, mapping.fieldMapping, or mapping.sortKeys; otherwise the connector cannot build the vertex or edge ID.

Schema Configuration (schema_config)

schema_config defines how one input stream is mapped to a specific vertex or edge label in HugeGraph.

Name	Type	Required	Default Value	Description
type	String	Yes	-	The type of graph element to map to. Must be VERTEX or EDGE.
label	String	Yes	-	The label of the vertex or edge in HugeGraph.
tablePath	String	No	-	Reserved table path value carried in the schema config.
properties	List<String>	No	-	A list of source field names for the vertex or edge.
ttl	Long	No	-	The time-to-live for the vertex or edge in seconds.
ttlStartTime	String	No	-	The start time for the TTL.
enableLabelIndex	String	No	-	Reserved label-index setting passed through the schema config.
userdata	Map<String, Object>	No	-	User-defined data associated with the label.
idStrategy	String	For Vertex	-	The ID generation strategy for vertices, such as PRIMARY_KEY, CUSTOMIZE_STRING, CUSTOMIZE_NUMBER, CUSTOMIZE_UUID, or AUTOMATIC.
idFields	List<string>	For Vertex	-	A list of source field names used to generate the vertex ID.
sourceConfig	Object	For Edge	-	An object defining the mapping for the edge's source vertex. See Source/Target Config below.
targetConfig	Object	For Edge	-	An object defining the mapping for the edge's target vertex. See Source/Target Config below.
frequency	String	For Edge	-	The frequency of the edge, e.g., SINGLE, MULTIPLE.
mapping	Object	No	-	An object defining advanced field and value mappings. See Mapping Config below.

Source/Target Config (sourceConfig and targetConfig)

This object is used within an EDGE schema to define how to identify the source and target vertices.

Name	Type	Required	Default Value	Description
label	String	Yes	-	The label of the source or target vertex.
idFields	List<String>	Yes	-	A list of source field names from the input row used to construct the ID of the source/target vertex. The values will be concatenated to form the vertex ID.

Mapping Config (mapping)

This object provides advanced control over how fields and values are mapped to properties.

Name	Type	Required	Default Value	Description
fieldMapping	Map<String, String>	No	-	A map where the key is the source field name and the value is the target property name in HugeGraph. If not specified, the source field name is used as the target property name.
valueMapping	Map<Object, Object>	No	-	A map to transform specific field values. The key is the original value from the source, and the value is the new value to be written.
nullableKeys	List<String>	No	-	A list of property keys that can have null values.
nullValues	List<String>	No	-	A list of string values that should be treated as null. Any field containing one of these values will not be written.
dateFormat	String	No	yyyy-MM-dd	The date format for parsing date strings.
timeZone	String	No	GMT+8	The time zone for date parsing.
sortKeys	List<String>	For Edge	-	A list of property keys to sort edges with the same source and target vertices.

Supported Types

The connector validates the SeaTunnel row schema against the existing HugeGraph schema before writing.

SeaTunnel type	HugeGraph property type
BYTES	BLOB
TINYINT	INT
SMALLINT	INT
INT	INT
BIGINT	LONG
FLOAT	FLOAT
DOUBLE	DOUBLE
BOOLEAN	BOOLEAN
DATE	DATE
TIMESTAMP	DATE
ARRAY	A non-single-cardinality HugeGraph property whose element type is compatible
STRING	TEXT
DECIMAL	TEXT
MAP	TEXT
ROW	TEXT
TIME	TEXT
NULL	TEXT

Write Behavior Notes

• For vertices, idStrategy controls how the vertex ID is built. PRIMARY_KEY joins all idFields with HugeGraph's primary-key format, CUSTOMIZE_STRING joins them with :, CUSTOMIZE_NUMBER expects one numeric field, and CUSTOMIZE_UUID expects one UUID field.
• For edges, the connector reads the ID strategy from the existing source and target vertex labels in HugeGraph. sourceConfig.idFields and targetConfig.idFields must provide enough fields to rebuild those vertex IDs.
• INSERT writes a new vertex or edge, UPDATE_AFTER updates the existing graph element, and DELETE deletes it. Delete rows only need the fields required to build the element ID.
• mapping.nullValues treats matching string values as null and skips those properties during writes.

Usage Examples

The examples below assume the corresponding HugeGraph schema has already been created.

1. Writing Vertices

This example shows how to read from a FakeSource and write person vertices to HugeGraph. The vertex ID is based on the name field.

env {
  job.mode = "BATCH"
}

source {
  FakeSource {
    plugin_input = "fake_source"
    schema = {
      fields = {
        name = "string"
        age = "int"
      }
    }
  }
}

sink {
  HugeGraph {
    host = "localhost"
    port = 8080
    graph_name = "hugegraph"
    graph_space = "default"
    selected_fields = ["name", "age"]
    schema_config = {
      type = "VERTEX"
      label = "person"
      idStrategy = "PRIMARY_KEY"
      idFields = ["name"]
      properties = ["name", "age"]
    }
  }
}

2. Writing Edges

This example syncs a relationship table to knows edges in HugeGraph. The source table contains the names of the two people who know each other and the year they met.

env {
  job.mode = "BATCH"
}

source {
  FakeSource {
    plugin_input = "fake_source"
    schema = {
      fields = {
        person1_name = "string"
        person2_name = "string"
        since = "int"
      }
    }
  }
}

sink {
  HugeGraph {
    host = "localhost"
    port = 8080
    graph_name = "hugegraph"
    graph_space = "default"
    schema_config = {
      type = "EDGE"
      label = "knows"
      sourceConfig = {
        label = "person"
        idFields = ["person1_name"]
      }
      targetConfig = {
        label = "person"
        idFields = ["person2_name"]
      }
      properties = ["since"]
      mapping = {
        fieldMapping = {
          person1_name = "name"
          person2_name = "name"
        }
      }
    }
  }
}

Changelog

Change Log

Change	Commit	Version
[Feature][Connector-V2] Support sink connector for Apache HugeGraph	https://github.com/apache/seatunnel/pull/10002/commits/002a653d11f48c3f76b47db23f5f2a68bc9d690c	2.3.12