Skip to main content
Version: Next

Gitlab

Gitlab source connector

Description​

The Gitlab source connector reads data from the GitLab REST API. It is built on the HTTP source connector, and automatically sends access_token as the GitLab PRIVATE-TOKEN request header.

Key features​

Source Options​

nametyperequireddefault value
urlStringYes-
access_tokenStringYes-
methodStringNoGET
headersMapNo-
paramsMapNo-
bodyStringNo-
formatStringNotext
schemaConfigNo-
schema.fieldsConfigNo-
json_fieldConfigNo-
content_fieldStringNo-
pageingConfigNo-
pageing.page_typeStringNoPageNumber
pageing.cursor_fieldStringNo-
pageing.cursor_response_fieldStringNo-
poll_interval_millisintNo-
retryintNo-
retry_backoff_multiplier_msintNo100
retry_backoff_max_msintNo10000
enable_multi_linesbooleanNofalse
keep_params_as_formbooleanNofalse
keep_page_param_as_http_parambooleanNofalse
pageing.batch_sizeintNo100
pageing.start_page_numberlongNo1
pageing.total_page_sizelongNo0
pageing.use_placeholder_replacementbooleanNofalse
connect_timeout_msintNo12000
socket_timeout_msintNo60000
json_filed_missed_return_nullbooleanNofalse
common-optionsconfigNo-

url [String]​

GitLab REST API URL, for example https://gitlab.com/api/v4/projects.

access_token [String]​

GitLab personal access token. The connector sends it in the HTTP PRIVATE-TOKEN header.

method [String]​

HTTP request method. The common GitLab read scenario uses GET.

headers [Map]​

Extra HTTP headers. Do not put PRIVATE-TOKEN here unless you intentionally want to override the header generated from access_token.

params [Map]​

HTTP query parameters, such as per_page, page, owned, or other GitLab API parameters.

body [String]​

HTTP request body. This is only useful for API endpoints that accept a request body.

format [String]​

Response format. Supports json and text. Use json with schema when you want SeaTunnel rows with named fields.

schema [Config]​

Defines the output row structure when format = "json". For details, see Schema Feature.

json_field [Config]​

Maps output fields to JSONPath expressions. Use it with schema when the required values are nested in the response.

content_field [String]​

JSONPath expression used to select a JSON fragment before schema parsing, for example $.items[*].

pageing [Config]​

Pagination settings inherited from the HTTP connector option rule. These options are not applied by this connector: GitlabSource builds its reader without a pagination context, so the source always reads a single page regardless of pageing settings, and the job still succeeds. Do not rely on pageing here; passing more data requires connector support to be added.

pageing.page_type [String]​

Pagination type. Supported values are PageNumber (default) and Cursor. These options describe how pagination would be configured, but as noted above they are not applied by this connector, so the source reads a single page regardless of this setting.

pageing.cursor_field [String]​

The request parameter name that carries the cursor value. Used together with pageing.page_type = "Cursor".

pageing.cursor_response_field [String]​

The JSONPath of the cursor in the response body. Used together with pageing.page_type = "Cursor".

poll_interval_millis [int]​

This option is inherited from the HTTP connector, but Gitlab source currently supports batch mode only.

retry [int]​

Maximum retry count when an HTTP request fails with IOException.

retry_backoff_multiplier_ms [int]​

Retry backoff multiplier in milliseconds.

retry_backoff_max_ms [int]​

Maximum retry backoff in milliseconds.

enable_multi_lines [boolean]​

When true, multiple JSON objects separated by newlines in the response body are treated as separate records.

keep_params_as_form [boolean]​

When true, request parameters are sent as form-encoded body parameters instead of URL query parameters.

keep_page_param_as_http_param [boolean]​

When true, the page parameter remains in the request URL when paginating instead of being replaced inside the body.

pageing.batch_size [int]​

The number of records returned per page request when the total number of pages is unknown.

pageing.start_page_number [long]​

Which page number to start synchronizing from.

pageing.total_page_size [long]​

Total page size to read. 0 means use pageing.batch_size until the API stops returning new pages.

pageing.use_placeholder_replacement [boolean]​

When true, use ${field} placeholder replacement for headers, parameters and body values; otherwise use key-based replacement.

connect_timeout_ms [int]​

HTTP connection timeout in milliseconds. Default 12000ms.

socket_timeout_ms [int]​

HTTP socket timeout in milliseconds. Default 60000ms.

json_filed_missed_return_null [boolean]​

When true, missing JSON fields return null; otherwise a missing field causes an error.

common options​

Source plugin common parameters. See Source Common Options.

Usage Notes​

  • access_token is sensitive. Avoid hardcoding real tokens in shared job files. Use SeaTunnel variable substitution or your deployment secret mechanism.
  • The connector always adds a PRIVATE-TOKEN header from access_token. Put other custom headers in headers.
  • Set format = "json" and define schema when you want typed SeaTunnel rows.
  • Use content_field when the GitLab response wraps records in a nested array such as $.items[*].
  • This connector currently reads a single page only: the pageing options are accepted by the option rule but not applied by the reader (see the pageing section). Do not configure params with page / per_page expecting the connector to walk pages.
  • The GitLab source supports batch mode only; poll_interval_millis does not enable streaming behavior.

Task Examples​

Read Projects From GitLab​

env {
parallelism = 1
job.mode = "BATCH"
}

source {
Gitlab {
url = "https://gitlab.com/api/v4/projects"
access_token = "glpat-xxxxxxxxxxxx"
method = "GET"
format = "json"
schema = {
fields {
id = int
description = string
name = string
name_with_namespace = string
path = string
http_url_to_repo = string
}
}
}
}

sink {
Console {
}
}

Read GitLab API Results (Single Page)​

env {
parallelism = 1
job.mode = "BATCH"
}

source {
Gitlab {
url = "https://gitlab.com/api/v4/projects"
access_token = "glpat-xxxxxxxxxxxx"
method = "GET"
params = {
per_page = "100"
page = "1"
}
format = "json"
schema = {
fields {
id = int
name = string
path = string
}
}
}
}

Filter And Extract Fields With JSONPath​

env {
parallelism = 1
job.mode = "BATCH"
}

source {
Gitlab {
url = "https://gitlab.com/api/v4/projects"
access_token = "glpat-xxxxxxxxxxxx"
method = "GET"
params = {
owned = "true"
per_page = "50"
}
format = "json"
content_field = "$.[*]"
json_field = {
id = "$.id"
name = "$.name"
visibility = "$.visibility"
}
schema = {
fields {
id = int
name = string
visibility = string
}
}
}
}

Changelog​

Change Log
ChangeCommitVersion
[improve] http connector options (#8969)https://github.com/apache/seatunnel/commit/63ff9f910a2.3.10
[Feature][Connector-V2] Support TableSourceFactory/TableSinkFactory on http (#5816)https://github.com/apache/seatunnel/commit/6f49ec6ead2.3.4
[Improve][build] Give the maven module a human readable name (#4114)https://github.com/apache/seatunnel/commit/d7cd6010512.3.1
[Improve][Project] Code format with spotless plugin. (#4101)https://github.com/apache/seatunnel/commit/a2ab1665612.3.1
[Improve][Connector-V2][Http]Improve json parse option rule for all http connector (#3627)https://github.com/apache/seatunnel/commit/589e4161ec2.3.0
[Feature][Connector-V2][HTTP] Use json-path parsing (#3510)https://github.com/apache/seatunnel/commit/1807eb6c952.3.0
[Hotfix][OptionRule] Fix option rule about all connectors (#3592)https://github.com/apache/seatunnel/commit/226dc6a1192.3.0
[Improve][Connector-V2][Gitlab] Unified excetion for Gitlab connector and improve optione rule (#3533)https://github.com/apache/seatunnel/commit/77f68f1eef2.3.0
[Feature][Connector V2] add gitlab source connector (#3408)https://github.com/apache/seatunnel/commit/545595c6d22.3.0