org.apache.beam.sdk.io.clickhouse

Class ClickHouseIO.Write<T>

java.lang.Object

org.apache.beam.sdk.transforms.PTransform<PCollection<T>,PDone>

org.apache.beam.sdk.io.clickhouse.ClickHouseIO.Write<T>

All Implemented Interfaces:

java.io.Serializable, HasDisplayData

Enclosing class:

ClickHouseIO

public abstract static class ClickHouseIO.Write<T>
extends PTransform<PCollection<T>,PDone>

A PTransform to write to ClickHouse.

See Also:

Serialized Form

Field Summary

Fields inherited from class org.apache.beam.sdk.transforms.PTransform

name

Constructor Summary

Constructors

Constructor and Description
Write()

Method Summary

Modifier and Type	Method and Description
PDone	expand(PCollection<T> input) Override this method to specify how this PTransform should be expanded on the given InputT.
abstract Duration	initialBackoff()
abstract @Nullable java.lang.Boolean	insertDeduplicate()
abstract @Nullable java.lang.Boolean	insertDistributedSync()
abstract @Nullable java.lang.Long	insertQuorum()
abstract java.lang.String	jdbcUrl()
abstract Duration	maxCumulativeBackoff()
abstract long	maxInsertBlockSize()
abstract int	maxRetries()
abstract java.util.Properties	properties()
abstract java.lang.String	table()
abstract @Nullable TableSchema	tableSchema()
ClickHouseIO.Write<T>	withInitialBackoff(Duration value) Set initial backoff duration.
ClickHouseIO.Write<T>	withInsertDeduplicate(java.lang.Boolean value) For INSERT queries in the replicated table, specifies that deduplication of inserting blocks should be performed.
ClickHouseIO.Write<T>	withInsertDistributedSync(@Nullable java.lang.Boolean value) If setting is enabled, insert query into distributed waits until data will be sent to all nodes in cluster.
ClickHouseIO.Write<T>	withInsertQuorum(@Nullable java.lang.Long value) For INSERT queries in the replicated table, wait writing for the specified number of replicas and linearize the addition of the data.
ClickHouseIO.Write<T>	withMaxCumulativeBackoff(Duration value) Limits total time spent in backoff.
ClickHouseIO.Write<T>	withMaxInsertBlockSize(long value) The maximum block size for insertion, if we control the creation of blocks for insertion.
ClickHouseIO.Write<T>	withMaxRetries(int value) Maximum number of retries per insert.
ClickHouseIO.Write<T>	withTableSchema(@Nullable TableSchema tableSchema) Set TableSchema.

Methods inherited from class org.apache.beam.sdk.transforms.PTransform

compose, compose, getAdditionalInputs, getDefaultOutputCoder, getDefaultOutputCoder, getDefaultOutputCoder, getKindString, getName, populateDisplayData, toString, validate

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Detail

Write

public Write()

Method Detail

jdbcUrl

public abstract java.lang.String jdbcUrl()

table

public abstract java.lang.String table()

properties

public abstract java.util.Properties properties()

maxInsertBlockSize

public abstract long maxInsertBlockSize()

maxRetries

public abstract int maxRetries()

maxCumulativeBackoff

public abstract Duration maxCumulativeBackoff()

initialBackoff

public abstract Duration initialBackoff()

tableSchema

public abstract @Nullable TableSchema tableSchema()

insertDistributedSync

public abstract @Nullable java.lang.Boolean insertDistributedSync()

insertQuorum

public abstract @Nullable java.lang.Long insertQuorum()

insertDeduplicate

public abstract @Nullable java.lang.Boolean insertDeduplicate()

expand

public PDone expand(PCollection<T> input)

Description copied from class: PTransform

Override this method to specify how this PTransform should be expanded on the given InputT.

NOTE: This method should not be called directly. Instead apply the PTransform should be applied to the InputT using the apply method.

Composite transforms, which are defined in terms of other transforms, should return the output of one of the composed transforms. Non-composite transforms, which do not apply any transforms internally, should return a new unbound output and register evaluators (via backend-specific registration methods).

Specified by:

expand in class PTransform<PCollection<T>,PDone>

withMaxInsertBlockSize

public ClickHouseIO.Write<T> withMaxInsertBlockSize(long value)

The maximum block size for insertion, if we control the creation of blocks for insertion.

Parameters:

value - number of rows

Returns:

a PTransform writing data to ClickHouse

See Also:

ClickHouse documentation

withInsertDistributedSync

public ClickHouseIO.Write<T> withInsertDistributedSync(@Nullable java.lang.Boolean value)

If setting is enabled, insert query into distributed waits until data will be sent to all nodes in cluster.

Parameters:

value - true to enable, null for server default

Returns:

a PTransform writing data to ClickHouse

withInsertQuorum

public ClickHouseIO.Write<T> withInsertQuorum(@Nullable java.lang.Long value)

For INSERT queries in the replicated table, wait writing for the specified number of replicas and linearize the addition of the data. 0 - disabled.

This setting is disabled in default server settings.

Parameters:

value - number of replicas, 0 for disabling, null for server default

Returns:

a PTransform writing data to ClickHouse

See Also:

ClickHouse documentation

withInsertDeduplicate

public ClickHouseIO.Write<T> withInsertDeduplicate(java.lang.Boolean value)

For INSERT queries in the replicated table, specifies that deduplication of inserting blocks should be performed.

Enabled by default. Shouldn't be disabled unless your input has duplicate blocks, and you don't want to deduplicate them.

Parameters:

value - true to enable, null for server default

Returns:

a PTransform writing data to ClickHouse

withMaxRetries

public ClickHouseIO.Write<T> withMaxRetries(int value)

Maximum number of retries per insert.

See FluentBackoff.withMaxRetries(int).

Parameters:

value - maximum number of retries

Returns:

a PTransform writing data to ClickHouse

withMaxCumulativeBackoff

public ClickHouseIO.Write<T> withMaxCumulativeBackoff(Duration value)

Limits total time spent in backoff.

See FluentBackoff.withMaxCumulativeBackoff(org.joda.time.Duration).

Parameters:

value - maximum duration

Returns:

a PTransform writing data to ClickHouse

withInitialBackoff

public ClickHouseIO.Write<T> withInitialBackoff(Duration value)

Set initial backoff duration.

See FluentBackoff.withInitialBackoff(org.joda.time.Duration).

Parameters:

value - initial duration

Returns:

a PTransform writing data to ClickHouse

withTableSchema

public ClickHouseIO.Write<T> withTableSchema(@Nullable TableSchema tableSchema)

Set TableSchema. If not set, then TableSchema will be fetched from clickhouse server itself

Parameters:

tableSchema - schema of Table in which rows are going to be inserted

Returns:

a PTransform writing data to ClickHouse