Skip to main content
Version: Next

Telemetry

Integrating Metrices through Prometheus-exports can better seamlessly connect to related monitoring platforms such as Prometheus and Grafana, improving the ability to monitor and alarm of the SeaTunnel cluster.

You can configure telemetry's configurations in the seatunnel.yaml file.

The following is an example declarative configuration.

seatunnel:
engine:
telemetry:
metric:
enabled: true # Whether open metrics export

Metrics

The metric text of prometheus,which get from http://{instanceHost}:5801/hazelcast/rest/instance/metrics.

The metric text of openMetrics,which get from http://{instanceHost}:5801/hazelcast/rest/instance/openmetrics.

Available metrics include the following categories.

Note: All metrics both have the same labelName cluster, that's value is the config of hazelcast.cluster-name.

Node Metrics

MetricNameTypeLabelsDESCRIPTION
cluster_infoGaugehazelcastVersion, the version of hazelcast. master, seatunnel master address.Cluster info
cluster_timeGaugehazelcastVersion, the version of hazelcast.Cluster time
node_countGauge-Cluster node total count
node_stateGaugeaddress, server instance address,for example: "127.0.0.1:5801"Whether is up of seatunnel node
hazelcast_executor_executedCountGaugetype, the type of executor, including: "async" "client" "clientBlocking" "clientQuery" "io" "offloadable" "scheduled" "system"The hazelcast executor executedCount of seatunnel cluster node
hazelcast_executor_isShutdownGaugetype, the type of executor, including: "async" "client" "clientBlocking" "clientQuery" "io" "offloadable" "scheduled" "system"The hazelcast executor isShutdown of seatunnel cluster node
hazelcast_executor_isTerminatedGaugetype, the type of executor, including: "async" "client" "clientBlocking" "clientQuery" "io" "offloadable" "scheduled" "system"The hazelcast executor isTerminated of seatunnel cluster node
hazelcast_executor_maxPoolSizeGaugetype, the type of executor, including: "async" "client" "clientBlocking" "clientQuery" "io" "offloadable" "scheduled" "system"The hazelcast executor maxPoolSize of seatunnel cluster node
hazelcast_executor_poolSizeGaugetype, the type of executor, including: "async" "client" "clientBlocking" "clientQuery" "io" "offloadable" "scheduled" "system"The hazelcast executor poolSize of seatunnel cluster node
hazelcast_executor_queueRemainingCapacityGaugetype, the type of executor, including: "async" "client" "clientBlocking" "clientQuery" "io" "offloadable" "scheduled" "system"The hazelcast executor queueRemainingCapacity of seatunnel cluster node
hazelcast_executor_queueSizeGaugetype, the type of executor, including: "async" "client" "clientBlocking" "clientQuery" "io" "offloadable" "scheduled" "system"The hazelcast executor queueSize of seatunnel cluster node
hazelcast_partition_partitionCountGauge-The partitionCount of seatunnel cluster node
hazelcast_partition_activePartitionGauge-The activePartition of seatunnel cluster node
hazelcast_partition_isClusterSafeGauge-Whether is cluster safe of partition
hazelcast_partition_isLocalMemberSafeGauge-Whether is local member safe of partition

Thread Pool Status

MetricNameTypeLabelsDESCRIPTION
job_thread_pool_activeCountGaugeaddress, server instance address,for example: "127.0.0.1:5801"The activeCount of seatunnel coordinator job's executor cached thread pool
job_thread_pool_corePoolSizeGaugeaddress, server instance address,for example: "127.0.0.1:5801"The corePoolSize of seatunnel coordinator job's executor cached thread pool
job_thread_pool_maximumPoolSizeGaugeaddress, server instance address,for example: "127.0.0.1:5801"The maximumPoolSize of seatunnel coordinator job's executor cached thread pool
job_thread_pool_poolSizeGaugeaddress, server instance address,for example: "127.0.0.1:5801"The poolSize of seatunnel coordinator job's executor cached thread pool
job_thread_pool_queueTaskCountGaugeaddress, server instance address,for example: "127.0.0.1:5801"The queueTaskCount of seatunnel coordinator job's executor cached thread pool
job_thread_pool_completedTask_totalCounteraddress, server instance address,for example: "127.0.0.1:5801"The completedTask of seatunnel coordinator job's executor cached thread pool
job_thread_pool_task_totalCounteraddress, server instance address,for example: "127.0.0.1:5801"The taskCount of seatunnel coordinator job's executor cached thread pool
job_thread_pool_rejection_totalCounteraddress, server instance address,for example: "127.0.0.1:5801"The rejectionCount of seatunnel coordinator job's executor cached thread pool

Job info detail

MetricNameTypeLabelsDESCRIPTION
job_countGaugetype, the type of job, including: "canceled" "cancelling" "created" "failed" "failing" "finished" "running" "scheduled"All job counts of seatunnel cluster

JVM Metrics

MetricNameTypeLabelsDESCRIPTION
jvm_threads_currentGauge-Current thread count of a JVM
jvm_threads_daemonGauge-Daemon thread count of a JVM
jvm_threads_peakGauge-Peak thread count of a JVM
jvm_threads_started_totalCounter-Started thread count of a JVM
jvm_threads_deadlockedGauge-Cycles of JVM-threads that are in deadlock waiting to acquire object monitors or ownable synchronizers
jvm_threads_deadlocked_monitorGauge-Cycles of JVM-threads that are in deadlock waiting to acquire object monitors
jvm_threads_stateGaugestate, the state of jvm thread, including: "NEW" "TERMINATED" "RUNNABLE" "BLOCKED" "WAITING" "TIMED_WAITING" "UNKNOWN"Current count of threads by state
jvm_classes_currently_loadedGauge-The number of classes that are currently loaded in the JVM
jvm_classes_loaded_totalCounter-The total number of classes that have been loaded since the JVM has started execution
jvm_classes_unloaded_totalCounter-The total number of classes that have been unloaded since the JVM has started execution
jvm_memory_pool_allocated_bytes_totalCounterpool,including: "Code Cache" "PS Eden Space" "PS Old Ge" "PS Survivor Space" "Compressed Class Space" "Metaspace"Total bytes allocated in a given JVM memory pool. Only updated after GC, not continuously
jvm_gc_collection_seconds_countSummarygc,including: "PS Scavenge" "PS MarkSweep"Time spent in a given JVM garbage collector in seconds
jvm_gc_collection_seconds_sumSummarygc,including: "PS Scavenge" "PS MarkSweep"Time spent in a given JVM garbage collector in seconds
jvm_infoGaugeruntime, for example: "Java(TM) SE Runtime Environment". vendor, for example: "Oracle Corporation". version ,for example: "1.8.0_212-b10"VM version info
process_cpu_seconds_totalCounter-Total user and system CPU time spent in seconds
process_start_time_secondsGauge-Start time of the process since unix epoch in seconds
process_open_fdsGauge-Number of open file descriptors
process_max_fdsGauge-Maximum number of open file descriptors
jvm_memory_objects_pending_finalizationGauge-The number of objects waiting in the finalizer queue
jvm_memory_bytes_usedGaugearea, including: "heap" "noheap"Used bytes of a given JVM memory area
jvm_memory_bytes_committedGaugearea, including: "heap" "noheap"Committed (bytes) of a given JVM memory area
jvm_memory_bytes_maxGaugearea, including:"heap" "noheap"Max (bytes) of a given JVM memory area
jvm_memory_bytes_initGaugearea, including:"heap" "noheap"Initial bytes of a given JVM memory area
jvm_memory_pool_bytes_usedGaugepool, including: "Code Cache" "PS Eden Space" "PS Old Ge" "PS Survivor Space" "Compressed Class Space" "Metaspace"Used bytes of a given JVM memory pool
jvm_memory_pool_bytes_committedGaugepool, including: "Code Cache" "PS Eden Space" "PS Old Ge" "PS Survivor Space" "Compressed Class Space" "Metaspace"Committed bytes of a given JVM memory pool
jvm_memory_pool_bytes_maxGaugepool, including: "Code Cache" "PS Eden Space" "PS Old Ge" "PS Survivor Space" "Compressed Class Space" "Metaspace"Max bytes of a given JVM memory pool
jvm_memory_pool_bytes_initGaugepool, including: "Code Cache" "PS Eden Space" "PS Old Ge" "PS Survivor Space" "Compressed Class Space" "Metaspace"Initial bytes of a given JVM memory pool
jvm_memory_pool_allocated_bytes_createdGaugepool, including: "Code Cache" "PS Eden Space" "PS Old Ge" "PS Survivor Space" "Compressed Class Space" "Metaspace"Total bytes allocated in a given JVM memory pool. Only updated after GC, not continuously
jvm_memory_pool_collection_used_bytesGaugepool, including: "PS Eden Space" "PS Old Ge" "PS Survivor Space"Used bytes after last collection of a given JVM memory pool
jvm_memory_pool_collection_committed_bytesGaugepool, including: "PS Eden Space" "PS Old Ge" "PS Survivor Space"Committed after last collection bytes of a given JVM memory pool
jvm_memory_pool_collection_max_bytesGaugepool, including: "PS Eden Space" "PS Old Ge" "PS Survivor Space"Max bytes after last collection of a given JVM memory pool
jvm_memory_pool_collection_init_bytesGaugepool, including: "PS Eden Space" "PS Old Ge" "PS Survivor Space"Initial after last collection bytes of a given JVM memory pool
jvm_buffer_pool_used_bytesGaugepool, including: "direct" "mapped"Used bytes of a given JVM buffer pool
jvm_buffer_pool_capacity_bytesGaugepool, including: "direct" "mapped"Bytes capacity of a given JVM buffer pool
jvm_buffer_pool_used_buffersGaugepool, including: "direct" "mapped"Used buffers of a given JVM buffer pool

Cluster Monitoring By Prometheus & Grafana

Install Prometheus

For a guide on how to set up Prometheus server go to the Installation

Configuration Prometheus

Add seatunnel instance metric exports into /etc/prometheus/prometheus.yaml. For example:

global:
# How frequently to scrape targets from this job.
scrape_interval: 15s
scrape_configs:
# The job name assigned to scraped metrics by default.
- job_name: 'seatunnel'
scrape_interval: 5s
# Metrics export path
metrics_path: /hazelcast/rest/instance/metrics
# List of labeled statically configured targets for this job.
static_configs:
# The targets specified by the static config.
- targets: [ 'localhost:5801' ]
# Labels assigned to all metrics scraped from the targets.
# labels: [<labelName>:<labelValue>]

Install Grafana

For a guide on how to set up Grafana server go to the Installation

Monitoring Dashboard

  • Add Prometheus DataSource on Grafana.
    • Import Seatunnel Cluster monitoring dashboard by Dashboard JSON into Grafana.

The effect image of the dashboard