org.apache.beam.sdk.io.jdbc

Class JdbcIO.WriteVoid<T>

java.lang.Object

org.apache.beam.sdk.transforms.PTransform<PCollection<T>,PCollection<java.lang.Void>>

org.apache.beam.sdk.io.jdbc.JdbcIO.WriteVoid<T>

All Implemented Interfaces:

java.io.Serializable, HasDisplayData

Enclosing class:

JdbcIO

public abstract static class JdbcIO.WriteVoid<T>
extends PTransform<PCollection<T>,PCollection<java.lang.Void>>

A PTransform to write to a JDBC datasource. Executes statements in a batch, and returns a trivial result.

See Also:

Serialized Form

Field Summary

Fields inherited from class org.apache.beam.sdk.transforms.PTransform

annotations, displayData, name, resourceHints

Constructor Summary

Constructors

Constructor and Description
WriteVoid()

Method Summary

Modifier and Type	Method and Description
PCollection<java.lang.Void>	expand(PCollection<T> input) Override this method to specify how this PTransform should be expanded on the given InputT.
JdbcIO.WriteVoid<T>	withAutoSharding() If true, enables using a dynamically determined number of shards to write.
JdbcIO.WriteVoid<T>	withBatchSize(long batchSize) Provide a maximum size in number of SQL statement for the batch.
JdbcIO.WriteVoid<T>	withDataSourceConfiguration(JdbcIO.DataSourceConfiguration config)
JdbcIO.WriteVoid<T>	withDataSourceProviderFn(SerializableFunction<java.lang.Void,javax.sql.DataSource> dataSourceProviderFn)
JdbcIO.WriteVoid<T>	withMaxBatchBufferingDuration(long maxBatchBufferingDuration) Provide maximum buffering time to batch elements before committing SQL statement.
JdbcIO.WriteVoid<T>	withPreparedStatementSetter(JdbcIO.PreparedStatementSetter<T> setter)
JdbcIO.WriteVoid<T>	withRetryConfiguration(JdbcIO.RetryConfiguration retryConfiguration) When a SQL exception occurs, JdbcIO.Write uses this JdbcIO.RetryConfiguration to exponentially back off and retry the statements based on the JdbcIO.RetryConfiguration mentioned.
JdbcIO.WriteVoid<T>	withRetryStrategy(JdbcIO.RetryStrategy retryStrategy) When a SQL exception occurs, JdbcIO.Write uses this JdbcIO.RetryStrategy to determine if it will retry the statements.
JdbcIO.WriteVoid<T>	withStatement(java.lang.String statement)
JdbcIO.WriteVoid<T>	withStatement(ValueProvider<java.lang.String> statement)
JdbcIO.WriteVoid<T>	withTable(java.lang.String table)

Methods inherited from class org.apache.beam.sdk.transforms.PTransform

addAnnotation, compose, compose, getAdditionalInputs, getAnnotations, getDefaultOutputCoder, getDefaultOutputCoder, getDefaultOutputCoder, getKindString, getName, getResourceHints, populateDisplayData, setDisplayData, setResourceHints, toString, validate, validate

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Detail

WriteVoid

public WriteVoid()

Method Detail

withAutoSharding

public JdbcIO.WriteVoid<T> withAutoSharding()

If true, enables using a dynamically determined number of shards to write.

withDataSourceConfiguration

public JdbcIO.WriteVoid<T> withDataSourceConfiguration(JdbcIO.DataSourceConfiguration config)

withDataSourceProviderFn

public JdbcIO.WriteVoid<T> withDataSourceProviderFn(SerializableFunction<java.lang.Void,javax.sql.DataSource> dataSourceProviderFn)

withStatement

public JdbcIO.WriteVoid<T> withStatement(java.lang.String statement)

withStatement

public JdbcIO.WriteVoid<T> withStatement(ValueProvider<java.lang.String> statement)

withPreparedStatementSetter

public JdbcIO.WriteVoid<T> withPreparedStatementSetter(JdbcIO.PreparedStatementSetter<T> setter)

withBatchSize

public JdbcIO.WriteVoid<T> withBatchSize(long batchSize)

Provide a maximum size in number of SQL statement for the batch. Default is 1000. The pipeline will either commit a batch when this maximum is reached or its maximum buffering time has been reached. See withMaxBatchBufferingDuration(long)

Parameters:

batchSize - maximum batch size in number of statements

withMaxBatchBufferingDuration

public JdbcIO.WriteVoid<T> withMaxBatchBufferingDuration(long maxBatchBufferingDuration)

Provide maximum buffering time to batch elements before committing SQL statement. Default is 200 The pipeline will either commit a batch when this maximum buffering time has been reached or the maximum amount of elements has been collected. See withBatchSize(long)

Parameters:

maxBatchBufferingDuration - maximum time in milliseconds before batch is committed

withRetryStrategy

public JdbcIO.WriteVoid<T> withRetryStrategy(JdbcIO.RetryStrategy retryStrategy)

When a SQL exception occurs, JdbcIO.Write uses this JdbcIO.RetryStrategy to determine if it will retry the statements. If JdbcIO.RetryStrategy.apply(SQLException) returns true, then JdbcIO.Write retries the statements.

withRetryConfiguration

public JdbcIO.WriteVoid<T> withRetryConfiguration(JdbcIO.RetryConfiguration retryConfiguration)

When a SQL exception occurs, JdbcIO.Write uses this JdbcIO.RetryConfiguration to exponentially back off and retry the statements based on the JdbcIO.RetryConfiguration mentioned.

Usage of RetryConfiguration -

 pipeline.apply(JdbcIO.<T>write())
    .withDataSourceConfiguration(...)
    .withRetryStrategy(...)
    .withRetryConfiguration(JdbcIO.RetryConfiguration.
        create(5, Duration.standardSeconds(5), Duration.standardSeconds(1))

 

maxDuration and initialDuration are Nullable

 pipeline.apply(JdbcIO.<T>write())
    .withDataSourceConfiguration(...)
    .withRetryStrategy(...)
    .withRetryConfiguration(JdbcIO.RetryConfiguration.
        create(5, null, null)

 

withTable

public JdbcIO.WriteVoid<T> withTable(java.lang.String table)

expand

public PCollection<java.lang.Void> expand(PCollection<T> input)

Description copied from class: PTransform

Override this method to specify how this PTransform should be expanded on the given InputT.

NOTE: This method should not be called directly. Instead apply the PTransform should be applied to the InputT using the apply method.

Composite transforms, which are defined in terms of other transforms, should return the output of one of the composed transforms. Non-composite transforms, which do not apply any transforms internally, should return a new unbound output and register evaluators (via backend-specific registration methods).

Specified by:

expand in class PTransform<PCollection<T>,PCollection<java.lang.Void>>