BigQuery
BigQuery source connector
Support Those Engines
Spark
Flink
Seatunnel Zeta
Key Features
Description
Source connector for Google Cloud BigQuery. Reads data by executing a SQL query and converting the results into SeaTunnel rows. The connector is a bounded source: it reads the query results and then finishes.
Supported DataSource Info
| Datasource | Supported Versions | Maven |
|---|---|---|
| BigQuery | 2.56.0 | Download |
Options
| Name | Type | Required | Default | Description |
|---|---|---|---|---|
| project_id | string | Yes | - | GCP project ID |
| query | string | No | - | SQL query to execute against BigQuery. Either query or the dataset_id + table_id pair is required. |
| dataset_id | string | No | - | BigQuery dataset ID. Used with table_id when query is not provided, or to set defaultDataset on the query job. |
| table_id | string | No | - | BigQuery table ID. Used with dataset_id when query is not provided. |
| service_account_key_path | string | No | - | Path to GCP service account JSON key file |
| service_account_key_json | string | No | - | Inline GCP service account JSON key content |
| max_results | long | No | 100 | BigQuery API page size. The SDK's TableResult handles pagination internally. |
| fetch_size | int | No | 0 | SeaTunnel internal batch size — max rows emitted per poll cycle. Zero means no limit. Independent of max_results. A smaller value allows more frequent checkpointing between batches. |
| location | string | No | - | BigQuery job location, set on JobId.setLocation(). Required for non-US/EU jobs. |
| use_query_cache | boolean | No | true | Whether to use BigQuery query result caching, mapped to QueryJobConfiguration.useQueryCache(). |
| dry_run | boolean | No | false | When true, the reader executes a dry run instead of the real query at runtime and emits no rows, mapped to QueryJobConfiguration.dryRun(). Planning always discovers the schema via a zero-cost dry run regardless of this option. |
| maximum_bytes_billed | long | No | - | Cost cap; the query fails if estimated bytes exceed this limit, preventing accidental full-table scans, mapped to QueryJobConfiguration.maximumBytesBilled(). |
| partition_filter | string | No | - | Partition filter predicate for direct table reads (dataset_id + table_id mode only). Rejected with an error if combined with query. Required when the table has requirePartitionFilter enabled. |
| emulator_host | string | No | - | BigQuery emulator REST host, such as localhost:9050. This option is intended for tests only. |
| universe_domain | string | No | - | The Google Cloud Universe Domain, such as s3nsapis.fr for S3NS sovereign cloud. |
| common-options | No | - | Source common options. See Source Common Options. |
Authentication Options
For production BigQuery jobs, provide one of the following authentication methods. Authentication is skipped only when emulator_host is configured for tests.
- service_account_key_path: Path to service account JSON file
- service_account_key_json: Inline JSON key content
- Default credentials: Uses application default credentials (ADC) if neither is specified
Query Modes
The source supports two ways to read data:
- Custom query: Set
queryto a SQLSELECTstatement. The connector executes it and converts the results. - Table read: Set
dataset_idandtable_id(withoutquery). The connector generatesSELECT * FROM \project.dataset.table`automatically. Optionally setpartition_filterto add aWHERE` clause for partition pruning.
Schema is inferred from the query results or table metadata. When dataset_id and table_id are provided, the connector reads table metadata via the SDK's bigQuery.getTable(). When only query is provided, the connector always discovers the schema via a zero-cost dry run without executing the query, extracting the schema from JobStatistics.QueryStatistics.getSchema() (populated by BigQuery only for dry-run jobs); this planning step is independent of the dry_run option. If the dry-run response has no schema and emulator_host is configured (the BigQuery emulator does not return a dry-run schema), the factory falls back to a max_results = 1 query that executes but returns at most one row, still providing the schema for downstream transforms; without emulator_host, planning fails fast instead of executing the query. The maximum_bytes_billed cost cap is applied to the fallback query and to the query executed at runtime — BigQuery fails a job whose estimate exceeds the cap; the planning dry run only logs a WARN when the estimate exceeds the cap, since dry-run jobs are free.
max_results
max_results controls the BigQuery API page size (number of rows per page request). The SDK's TableResult handles pagination internally via getNextPage() / getNextPageToken(), so all rows are returned regardless of this setting. A larger value reduces the number of API round-trips for large results; a smaller value reduces memory pressure per page.
fetch_size
fetch_size controls SeaTunnel's internal batching: it is the maximum number of rows emitted to the downstream per pollNext cycle. A value of 0 (default) means no limit — all available rows are emitted in a single cycle. A non-zero value splits the output into batches of the given size, which allows more frequent checkpointing between batches.
This is independent of max_results: max_results controls the BigQuery HTTP API page size (how many rows the SDK fetches per request), while fetch_size controls how many rows SeaTunnel emits before yielding control back to the engine. A common pattern is to set max_results to a large value (to reduce API round-trips) and fetch_size to a smaller value (to allow checkpointing between batches).
location
location is set on JobId.setLocation() and tells BigQuery which region to run the job in. This is required for jobs that target datasets in non-US/EU regions (e.g., asia-northeast1, us-central1).
dry_run
When dry_run = true, the reader executes a dry-run job instead of the real query at runtime: it logs estimated bytes processed and signals noMoreElement immediately, and no rows are emitted. The estimated bytes are logged at INFO level via JobStatistics.QueryStatistics.getEstimatedBytesProcessed(). If maximum_bytes_billed is also set and the estimate exceeds it, a WARN is logged. When dry_run = false (default), the query is executed normally.
Schema discovery at planning is not affected by this option: in query mode the factory always performs a zero-cost dry run to validate the query and extract the schema from JobStatistics.QueryStatistics.getSchema() without executing it, so planning never bills scanned bytes regardless of dry_run. See Query Modes for details.
maximum_bytes_billed
maximum_bytes_billed sets a cost cap on the query. If the estimated bytes billed would exceed this limit, the query fails without executing. This prevents accidental full-table scans from incurring unexpected costs.
partition_filter
partition_filter is an optional predicate applied only in table-read mode (dataset_id + table_id). When provided, the connector appends it as a WHERE clause to the generated query:
SELECT * FROM `project.dataset.table` WHERE <partition_filter>
This enables partition pruning, which significantly reduces scan cost ($6.25/TB on-demand) for large partitioned tables. Example: partition_filter = "_PARTITIONDATE >= '2026-01-01'".
partition_filter is a raw SQL predicate appended directly to the generated query, at the same trust level as the query option. Both are administrator-provided configuration values.
Validation rules:
- Rejected with an error if combined with
querymode — include the filter directly in the SQL instead. - If the target table has
requirePartitionFilter = truein BigQuery metadata, the factory fails fast at validation time with a clear error before any job is submitted.
Task Example
Simple Table Read
env {
parallelism = 1
job.mode = "BATCH"
}
source {
BigQuery {
project_id = "my-gcp-project"
dataset_id = "public_data"
table_id = "orders"
service_account_key_path = "/path/to/key.json"
}
}
sink {
Console {}
}
Custom Query
env {
parallelism = 1
job.mode = "BATCH"
}
source {
BigQuery {
project_id = "my-gcp-project"
query = "SELECT order_id, customer_name, amount FROM `my-gcp-project.public_data.orders` WHERE amount > 100 ORDER BY order_id"
service_account_key_path = "/path/to/key.json"
max_results = 500
use_query_cache = true
}
}
sink {
Console {}
}
Dry Run for Cost Estimation
env {
parallelism = 1
job.mode = "BATCH"
}
source {
BigQuery {
project_id = "my-gcp-project"
query = "SELECT * FROM `my-gcp-project.public_data.large_table`"
service_account_key_path = "/path/to/key.json"
dry_run = true
maximum_bytes_billed = 1000000000
}
}
sink {
Console {}
}
With Location and Inline Key
env {
parallelism = 1
job.mode = "BATCH"
}
source {
BigQuery {
project_id = "my-gcp-project"
dataset_id = "analytics"
table_id = "events"
service_account_key_json = "${GCP_SA_KEY_JSON}"
location = "asia-northeast1"
}
}
sink {
Console {}
}
Note: service_account_key_json = "${GCP_SA_KEY_JSON}" uses SeaTunnel variable substitution — provide the variable value when submitting the job; if it is not provided, the raw ${...} literal is passed through unchanged.
Lifecycle and Delivery Semantics
The BigQuery source is a bounded source (Boundedness.BOUNDED):
- Split enumeration: The
BigQuerySourceSplitEnumeratorcreates a single split containing the resolved query and assigns it to the first reader. Readers that receive no split are sent a no-more-splits signal so idle readers can terminate. There is no parallel read — the BigQuery SDK handles pagination internally. - Query execution: On the first
pollNext, the reader callsBigQuery.query(QueryJobConfiguration)to execute the query. The SDK paginates results viaTableResult.iterateAll(). - Batching: If
fetch_size > 0, rows are emitted in batches offetch_sizeperpollNextcycle; otherwise all rows are emitted in a single cycle. Between batches, the engine can perform checkpointing. - Job cleanup: The BigQuery client is created by
BigQueryClientFactory.getBigQuery()each time it is called (not a singleton). Query jobs created byBigQuery.query()are automatically cleaned up by BigQuery after completion. The reader'sclose()is a no-op. - Delivery semantics: At-least-once. If a task fails mid-read, the query is re-executed on recovery. There is no offset tracking — the entire query result is re-read.
- Snapshot state:
snapshotState()returns the list of pending splits plus the in-flight split (the one currently being read), so that recovery re-executes the entire query from the current split. Since there is only one split, the snapshot is either empty (split fully consumed) or contains the single split.
Scope: Bounded-Batch Only (No CDC)
This connector is strictly bounded-batch. BigQuery does not expose a transaction log or WAL socket for streaming change data capture, and attempting CDC natively would cause high slot contention and quota exhaustion. For users who need streaming change capture out of BigQuery, Google's recommended architecture is BigQuery Continuous Queries → Cloud Pub/Sub → SeaTunnel's existing GooglePubSub Source.
Single Split and Future Parallelism
The current implementation uses a single split built on REST query pagination (max_results). This is sufficient for standard query and table export workloads. A future enhancement could introduce parallel multi-split reads via the BigQuery Storage Read API (ReadSession), but that is out of scope for this version and would require an additional third-party dependency.
Changelog
Change Log
| Change | Commit | Version |
|---|