Skip to main content
Version: Next

Github

Github source connector

Description​

The Github source connector reads data from the GitHub REST API. It is built on the HTTP source connector, and automatically adds the Authorization: Bearer <access_token> request header.

Key features​

Source Options​

nametyperequireddefault value
urlStringYes-
access_tokenStringYes-
methodStringNoGET
headersMapNo-
paramsMapNo-
bodyStringNo-
formatStringNotext
schemaConfigNo-
schema.fieldsConfigNo-
json_fieldConfigNo-
content_fieldStringNo-
pageingConfigNo-
pageing.page_typeStringNoPageNumber
pageing.cursor_fieldStringNo-
pageing.cursor_response_fieldStringNo-
poll_interval_millisintNo-
retryintNo-
retry_backoff_multiplier_msintNo100
retry_backoff_max_msintNo10000
enable_multi_linesbooleanNofalse
keep_params_as_formbooleanNofalse
keep_page_param_as_http_parambooleanNofalse
pageing.batch_sizeintNo100
pageing.start_page_numberlongNo1
pageing.total_page_sizelongNo0
pageing.use_placeholder_replacementbooleanNofalse
connect_timeout_msintNo12000
socket_timeout_msintNo60000
json_filed_missed_return_nullbooleanNofalse
common-optionsconfigNo-

url [String]​

GitHub REST API URL, for example https://api.github.com/orgs/apache/repos.

access_token [String]​

GitHub personal access token. The connector sends it as a Bearer token in the HTTP Authorization header.

method [String]​

HTTP request method. The common GitHub read scenario uses GET.

headers [Map]​

Extra HTTP headers. Do not put Authorization here unless you intentionally want to override the header generated from access_token.

params [Map]​

HTTP query parameters, such as per_page, page, since, or other GitHub API parameters.

body [String]​

HTTP request body. This is only useful for API endpoints that accept a request body.

format [String]​

Response format. Supports json and text. Use json with schema when you want SeaTunnel rows with named fields.

schema [Config]​

Defines the output row structure when format = "json". For details, see Schema Feature.

json_field [Config]​

Maps output fields to JSONPath expressions. Use it with schema when the required values are nested in the response.

content_field [String]​

JSONPath expression used to select a JSON fragment before schema parsing, for example $.items[*].

pageing [Config]​

Pagination settings inherited from the HTTP connector option rule. These options are not applied by this connector: GithubSource builds its reader without a pagination context, so the source always reads a single page regardless of pageing settings, and the job still succeeds. Do not rely on pageing here; passing more data requires connector support to be added.

pageing.page_type [String]​

Pagination type. Supported values are PageNumber (default) and Cursor. These options describe how pagination would be configured, but as noted above they are not applied by this connector, so the source reads a single page regardless of this setting.

pageing.cursor_field [String]​

The request parameter name that carries the cursor value. Used together with pageing.page_type = "Cursor".

pageing.cursor_response_field [String]​

The JSONPath of the cursor in the response body. Used together with pageing.page_type = "Cursor".

poll_interval_millis [int]​

Request interval in milliseconds for streaming jobs. In batch jobs the connector reads once and finishes.

retry [int]​

Maximum retry count when an HTTP request fails with IOException.

retry_backoff_multiplier_ms [int]​

Retry backoff multiplier in milliseconds.

retry_backoff_max_ms [int]​

Maximum retry backoff in milliseconds.

enable_multi_lines [boolean]​

When true, multiple JSON objects separated by newlines in the response body are treated as separate records.

keep_params_as_form [boolean]​

When true, request parameters are sent as form-encoded body parameters instead of URL query parameters.

keep_page_param_as_http_param [boolean]​

When true, the page parameter remains in the request URL when paginating instead of being replaced inside the body.

pageing.batch_size [int]​

The number of records returned per page request when the total number of pages is unknown.

pageing.start_page_number [long]​

Which page number to start synchronizing from.

pageing.total_page_size [long]​

Total page size to read. 0 means use pageing.batch_size until the API stops returning new pages.

pageing.use_placeholder_replacement [boolean]​

When true, use ${field} placeholder replacement for headers, parameters and body values; otherwise use key-based replacement.

connect_timeout_ms [int]​

HTTP connection timeout in milliseconds. Default 12000ms.

socket_timeout_ms [int]​

HTTP socket timeout in milliseconds. Default 60000ms.

json_filed_missed_return_null [boolean]​

When true, missing JSON fields return null; otherwise a missing field causes an error.

common options​

Source plugin common parameters. See Source Common Options.

Usage Notes​

  • access_token is sensitive. Avoid hardcoding real tokens in shared job files. Use SeaTunnel variable substitution or your deployment secret mechanism.
  • The connector always adds an Authorization: Bearer <access_token> header from access_token. Put other custom headers in headers.
  • Set format = "json" and define schema when you want typed SeaTunnel rows.
  • Use content_field when the GitHub response wraps records in a nested array such as $.items[*].
  • This connector currently reads a single page only: the pageing options are accepted by the option rule but not applied by the reader (see the pageing section). Do not configure params with page / per_page expecting the connector to walk pages.

Task Examples​

Read Repositories From A GitHub Organization​

env {
parallelism = 1
job.mode = "BATCH"
}

source {
Github {
url = "https://api.github.com/orgs/apache/repos"
access_token = "ghp_xxxxxxxxxxxx"
method = "GET"
format = "json"
schema = {
fields {
id = int
name = string
description = string
html_url = string
stargazers_count = int
forks = int
}
}
}
}

sink {
Console {
}
}

Read GitHub API Results (Single Page)​

env {
parallelism = 1
job.mode = "BATCH"
}

source {
Github {
url = "https://api.github.com/orgs/apache/repos"
access_token = "ghp_xxxxxxxxxxxx"
method = "GET"
params = {
per_page = "100"
page = "1"
}
format = "json"
schema = {
fields {
id = int
name = string
html_url = string
}
}
}
}

Stream GitHub Events​

env {
parallelism = 1
job.mode = "STREAMING"
checkpoint.interval = 30000
}

source {
Github {
url = "https://api.github.com/orgs/apache/events"
access_token = "ghp_xxxxxxxxxxxx"
method = "GET"
format = "json"
poll_interval_millis = 60000
schema = {
fields {
id = string
type = string
created_at = string
}
}
}
}

Changelog​

Change Log
ChangeCommitVersion
[improve] http connector options (#8969)https://github.com/apache/seatunnel/commit/63ff9f910a2.3.10
[Feature][Connector-V2] Support TableSourceFactory/TableSinkFactory on http (#5816)https://github.com/apache/seatunnel/commit/6f49ec6ead2.3.4
[Feature][Connector-V2][Github] Adding Github Source Connector (#4155)https://github.com/apache/seatunnel/commit/49d9172b102.3.1