Http

Http source connector

Description

Used to read data from Http.

Key features

• batch
• stream
• exactly-once
• column projection
• parallelism
• support user-defined split

Options

name	type	required	default value
url	String	Yes	-
schema	Config	No	-
schema.fields	Config	No	-
json_field	Config	No	-
content_json	String	No	-
format	String	No	json
method	String	No	get
headers	Map	No	-
params	Map	No	-
body	String	No	-
poll_interval_ms	int	No	-
retry	int	No	-
retry_backoff_multiplier_ms	int	No	100
retry_backoff_max_ms	int	No	10000
enable_multi_lines	boolean	No	false
common-options	config	No	-

url [String]

http request url

method [String]

http request method, only supports GET, POST method.

headers [Map]

http headers

params [Map]

http params

body [String]

http body

poll_interval_ms [int]

request http api interval(millis) in stream mode

retry [int]

The max retry times if request http return to IOException

retry_backoff_multiplier_ms [int]

The retry-backoff times(millis) multiplier if request http failed

retry_backoff_max_ms [int]

The maximum retry-backoff times(millis) if request http failed

format [String]

the format of upstream data, now only support json text, default json.

when you assign format is json, you should also assign schema option, for example:

upstream data is the following:

{
  "code": 200,
  "data": "get success",
  "success": true
}

you should assign schema as the following:

schema {
  fields {
    code = int
    data = string
    success = boolean
  }
}

connector will generate data as the following:

code	data	success
200	get success	true

when you assign format is text, connector will do nothing for upstream data, for example:

upstream data is the following:

{
  "code": 200,
  "data": "get success",
  "success": true
}

connector will generate data as the following:

content
{"code": 200, "data": "get success", "success": true}

schema [Config]

fields [Config]

the schema fields of upstream data

content_json [String]

This parameter can get some json data.If you only need the data in the 'book' section, configure content_field = "$.store.book.*".

If your return data looks something like this.

{
  "store": {
    "book": [
      {
        "category": "reference",
        "author": "Nigel Rees",
        "title": "Sayings of the Century",
        "price": 8.95
      },
      {
        "category": "fiction",
        "author": "Evelyn Waugh",
        "title": "Sword of Honour",
        "price": 12.99
      }
    ],
    "bicycle": {
      "color": "red",
      "price": 19.95
    }
  },
  "expensive": 10
}

You can configure content_field = "$.store.book.*" and the result returned looks like this:

[
  {
    "category": "reference",
    "author": "Nigel Rees",
    "title": "Sayings of the Century",
    "price": 8.95
  },
  {
    "category": "fiction",
    "author": "Evelyn Waugh",
    "title": "Sword of Honour",
    "price": 12.99
  }
]

Then you can get the desired result with a simpler schema,like

Http {
  url = "http://mockserver:1080/contentjson/mock"
  method = "GET"
  format = "json"
  content_field = "$.store.book.*"
  schema = {
    fields {
      category = string
      author = string
      title = string
      price = string
    }
  }
}

Here is an example:

• Test data can be found at this link mockserver-config.json
• See this link for task configuration http_contentjson_to_assert.conf.

json_field [Config]

This parameter helps you configure the schema,so this parameter must be used with schema.

If your data looks something like this:

{ 
  "store": {
    "book": [
      {
        "category": "reference",
        "author": "Nigel Rees",
        "title": "Sayings of the Century",
        "price": 8.95
      },
      {
        "category": "fiction",
        "author": "Evelyn Waugh",
        "title": "Sword of Honour",
        "price": 12.99
      }
    ],
    "bicycle": {
      "color": "red",
      "price": 19.95
    }
  },
  "expensive": 10
}

You can get the contents of 'book' by configuring the task as follows:

source {
  Http {
    url = "http://mockserver:1080/jsonpath/mock"
    method = "GET"
    format = "json"
    json_field = {
      category = "$.store.book[*].category"
      author = "$.store.book[*].author"
      title = "$.store.book[*].title"
      price = "$.store.book[*].price"
    }
    schema = {
      fields {
        category = string
        author = string
        title = string
        price = string
      }
    }
  }
}

• Test data can be found at this link mockserver-config.json
• See this link for task configuration http_jsonpath_to_assert.conf.

common options

Source plugin common parameters, please refer to Source Common Options for details

Example

simple:

Http {
  url = "https://tyrantlucifer.com/api/getDemoData"
  schema {
    fields {
      code = int
      message = string
      data = string
      ok = boolean
    }
  }
}

Changelog

2.2.0-beta 2022-09-26

• Add Http Source Connector

new version

• [Feature][Connector-V2][HTTP] Use json-path parsing (3510)