org.apache.beam.sdk.io.gcp.bigtable

Class BigtableIO.Write

java.lang.Object

org.apache.beam.sdk.transforms.PTransform<PCollection<KV<com.google.protobuf.ByteString,java.lang.Iterable<com.google.bigtable.v2.Mutation>>>,PDone>

org.apache.beam.sdk.io.gcp.bigtable.BigtableIO.Write

All Implemented Interfaces:

java.io.Serializable, HasDisplayData

Enclosing class:

BigtableIO

public abstract static class BigtableIO.Write
extends PTransform<PCollection<KV<com.google.protobuf.ByteString,java.lang.Iterable<com.google.bigtable.v2.Mutation>>>,PDone>

A PTransform that writes to Google Cloud Bigtable. See the class-level Javadoc on BigtableIO for more information.

See Also:

BigtableIO, Serialized Form

Field Summary

Fields inherited from class org.apache.beam.sdk.transforms.PTransform

annotations, displayData, name, resourceHints

Constructor Summary

Constructors

Constructor and Description
Write()

Method Summary

Modifier and Type	Method and Description
PDone	expand(PCollection<KV<com.google.protobuf.ByteString,java.lang.Iterable<com.google.bigtable.v2.Mutation>>> input) Override this method to specify how this PTransform should be expanded on the given InputT.
@Nullable BigtableOptions	getBigtableOptions() Deprecated. write options are configured directly on BigtableIO.write(). Use populateDisplayData(DisplayData.Builder) to view the current configurations.
void	populateDisplayData(DisplayData.Builder builder) Register display data for the given transform or component.
java.lang.String	toString()
void	validate(PipelineOptions options) Called before running the Pipeline to verify this transform is fully and correctly specified.
BigtableIO.Write	withAppProfileId(java.lang.String appProfileId) Returns a new BigtableIO.Write that will write using the specified app profile id.
BigtableIO.Write	withAppProfileId(ValueProvider<java.lang.String> appProfileId) Returns a new BigtableIO.Write that will write using the specified app profile id.
BigtableIO.Write	withAttemptTimeout(Duration timeout) Returns a new BigtableIO.Write with the attempt timeout.
BigtableIO.Write	withBigtableOptions(BigtableOptions.Builder optionsBuilder) Deprecated. please configure the write options directly: BigtableIO.write().withProjectId(projectId).withInstanceId(instanceId).withTableId(tableId) and set credentials in PipelineOptions.
BigtableIO.Write	withBigtableOptions(BigtableOptions options) Deprecated. please configure the write options directly: BigtableIO.write().withProjectId(projectId).withInstanceId(instanceId).withTableId(tableId) and set credentials in PipelineOptions.
BigtableIO.Write	withBigtableOptionsConfigurator(SerializableFunction<BigtableOptions.Builder,BigtableOptions.Builder> configurator) Deprecated. please configure the write options directly: BigtableIO.write().withProjectId(projectId).withInstanceId(instanceId).withTableId(tableId) and set credentials in PipelineOptions.
BigtableIO.Write	withEmulator(java.lang.String emulatorHost) Returns a new BigtableIO.Write that will use an official Bigtable emulator.
BigtableIO.Write	withErrorHandler(ErrorHandler<BadRecord,?> badRecordErrorHandler)
BigtableIO.Write	withFlowControl(boolean enableFlowControl) Returns a new BigtableIO.Write with flow control enabled if enableFlowControl is true.
BigtableIO.Write	withInstanceId(java.lang.String instanceId) Returns a new BigtableIO.Write that will write into the Cloud Bigtable instance indicated by given parameter, requires withProjectId(org.apache.beam.sdk.options.ValueProvider<java.lang.String>) to be called to determine the project.
BigtableIO.Write	withInstanceId(ValueProvider<java.lang.String> instanceId) Returns a new BigtableIO.Write that will write into the Cloud Bigtable instance indicated by given parameter, requires withProjectId(org.apache.beam.sdk.options.ValueProvider<java.lang.String>) to be called to determine the project.
BigtableIO.Write	withMaxBytesPerBatch(long size) Returns a new BigtableIO.Write with the max bytes a batch can have.
BigtableIO.Write	withMaxElementsPerBatch(long size) Returns a new BigtableIO.Write with the max elements a batch can have.
BigtableIO.Write	withMaxOutstandingBytes(long bytes) Returns a new BigtableIO.Write with the max number of outstanding bytes allowed before enforcing flow control.
BigtableIO.Write	withMaxOutstandingElements(long count) Returns a new BigtableIO.Write with the max number of outstanding elements allowed before enforcing flow control.
BigtableIO.Write	withOperationTimeout(Duration timeout) Returns a new BigtableIO.Write with the operation timeout.
BigtableIO.Write	withoutValidation() Disables validation that the table being written to exists.
BigtableIO.Write	withProjectId(java.lang.String projectId) Returns a new BigtableIO.Write that will write into the Cloud Bigtable project indicated by given parameter, requires withInstanceId(org.apache.beam.sdk.options.ValueProvider<java.lang.String>) to be called to determine the instance.
BigtableIO.Write	withProjectId(ValueProvider<java.lang.String> projectId) Returns a new BigtableIO.Write that will write into the Cloud Bigtable project indicated by given parameter, requires withInstanceId(org.apache.beam.sdk.options.ValueProvider<java.lang.String>) to be called to determine the instance.
BigtableIO.Write	withTableId(java.lang.String tableId) Returns a new BigtableIO.Write that will write to the specified table.
BigtableIO.Write	withTableId(ValueProvider<java.lang.String> tableId) Returns a new BigtableIO.Write that will write to the specified table.
BigtableIO.Write	withThrottlingReportTargetMs(int throttlingReportTargetMs) Deprecated. This method has been deprecated in Beam 2.60.0. It does not have an effect.
BigtableIO.Write	withThrottlingTargetMs(int throttlingTargetMs) Deprecated. This method has been deprecated in Beam 2.60.0. It does not have an effect.
BigtableIO.WriteWithResults	withWriteResults() Returns a BigtableIO.WriteWithResults that will emit a BigtableWriteResult for each batch of rows written.

Methods inherited from class org.apache.beam.sdk.transforms.PTransform

addAnnotation, compose, compose, getAdditionalInputs, getAnnotations, getDefaultOutputCoder, getDefaultOutputCoder, getDefaultOutputCoder, getKindString, getName, getResourceHints, setDisplayData, setResourceHints, validate

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Detail

Write

public Write()

Method Detail

getBigtableOptions

@Deprecated
public @Nullable BigtableOptions getBigtableOptions()

Deprecated. write options are configured directly on BigtableIO.write(). Use populateDisplayData(DisplayData.Builder) to view the current configurations.

Returns the Google Cloud Bigtable instance being written to, and other parameters.

withProjectId

public BigtableIO.Write withProjectId(ValueProvider<java.lang.String> projectId)

Returns a new BigtableIO.Write that will write into the Cloud Bigtable project indicated by given parameter, requires withInstanceId(org.apache.beam.sdk.options.ValueProvider<java.lang.String>) to be called to determine the instance.

Does not modify this object.

withProjectId

public BigtableIO.Write withProjectId(java.lang.String projectId)

Returns a new BigtableIO.Write that will write into the Cloud Bigtable project indicated by given parameter, requires withInstanceId(org.apache.beam.sdk.options.ValueProvider<java.lang.String>) to be called to determine the instance.

Does not modify this object.

withInstanceId

public BigtableIO.Write withInstanceId(ValueProvider<java.lang.String> instanceId)

Returns a new BigtableIO.Write that will write into the Cloud Bigtable instance indicated by given parameter, requires withProjectId(org.apache.beam.sdk.options.ValueProvider<java.lang.String>) to be called to determine the project.

Does not modify this object.

withInstanceId

public BigtableIO.Write withInstanceId(java.lang.String instanceId)

Returns a new BigtableIO.Write that will write into the Cloud Bigtable instance indicated by given parameter, requires withProjectId(org.apache.beam.sdk.options.ValueProvider<java.lang.String>) to be called to determine the project.

Does not modify this object.

withTableId

public BigtableIO.Write withTableId(ValueProvider<java.lang.String> tableId)

Returns a new BigtableIO.Write that will write to the specified table.

Does not modify this object.

withTableId

public BigtableIO.Write withTableId(java.lang.String tableId)

Returns a new BigtableIO.Write that will write to the specified table.

Does not modify this object.

withAppProfileId

public BigtableIO.Write withAppProfileId(ValueProvider<java.lang.String> appProfileId)

Returns a new BigtableIO.Write that will write using the specified app profile id.

Remember that in order to use single-row transactions, this must use a single-cluster routing policy.

Does not modify this object.

withAppProfileId

public BigtableIO.Write withAppProfileId(java.lang.String appProfileId)

Returns a new BigtableIO.Write that will write using the specified app profile id.

Remember that in order to use single-row transactions, this must use a single-cluster routing policy.

Does not modify this object.

withBigtableOptions

@Deprecated
public BigtableIO.Write withBigtableOptions(BigtableOptions options)

Deprecated. please configure the write options directly: BigtableIO.write().withProjectId(projectId).withInstanceId(instanceId).withTableId(tableId) and set credentials in PipelineOptions.

WARNING: Should be used only to specify additional parameters for connection to the Cloud Bigtable, instanceId and projectId should be provided over withInstanceId(org.apache.beam.sdk.options.ValueProvider<java.lang.String>) and withProjectId(org.apache.beam.sdk.options.ValueProvider<java.lang.String>) respectively.

Returns a new BigtableIO.Write that will write to the Cloud Bigtable instance indicated by the given options, and using any other specified customizations.

Does not modify this object.

withBigtableOptions

@Deprecated
public BigtableIO.Write withBigtableOptions(BigtableOptions.Builder optionsBuilder)

Deprecated. please configure the write options directly: BigtableIO.write().withProjectId(projectId).withInstanceId(instanceId).withTableId(tableId) and set credentials in PipelineOptions.

WARNING: Should be used only to specify additional parameters for connection to the Cloud Bigtable, instanceId and projectId should be provided over withInstanceId(org.apache.beam.sdk.options.ValueProvider<java.lang.String>) and withProjectId(org.apache.beam.sdk.options.ValueProvider<java.lang.String>) respectively.

Returns a new BigtableIO.Write that will write to the Cloud Bigtable instance indicated by the given options, and using any other specified customizations.

Clones the given BigtableOptions builder so that any further changes will have no effect on the returned BigtableIO.Write.

Does not modify this object.

withBigtableOptionsConfigurator

@Deprecated
public BigtableIO.Write withBigtableOptionsConfigurator(SerializableFunction<BigtableOptions.Builder,BigtableOptions.Builder> configurator)

Deprecated. please configure the write options directly: BigtableIO.write().withProjectId(projectId).withInstanceId(instanceId).withTableId(tableId) and set credentials in PipelineOptions.

Returns a new BigtableIO.Write that will read from the Cloud Bigtable instance with customized options provided by given configurator.

WARNING: instanceId and projectId should not be provided here and should be provided over withProjectId(org.apache.beam.sdk.options.ValueProvider<java.lang.String>) and withInstanceId(org.apache.beam.sdk.options.ValueProvider<java.lang.String>).

Does not modify this object.

withoutValidation

public BigtableIO.Write withoutValidation()

Disables validation that the table being written to exists.

withEmulator

public BigtableIO.Write withEmulator(java.lang.String emulatorHost)

Returns a new BigtableIO.Write that will use an official Bigtable emulator.

This is used for testing.

withAttemptTimeout

public BigtableIO.Write withAttemptTimeout(Duration timeout)

Returns a new BigtableIO.Write with the attempt timeout. Attempt timeout controls the timeout for each remote call.

Does not modify this object.

withOperationTimeout

public BigtableIO.Write withOperationTimeout(Duration timeout)

Returns a new BigtableIO.Write with the operation timeout. Operation timeout has ultimate control over how long the logic should keep trying the remote call until it gives up completely.

Does not modify this object.

withMaxElementsPerBatch

public BigtableIO.Write withMaxElementsPerBatch(long size)

Returns a new BigtableIO.Write with the max elements a batch can have. After this many elements are accumulated, they will be wrapped up in a batch and sent to Bigtable.

Does not modify this object.

withMaxBytesPerBatch

public BigtableIO.Write withMaxBytesPerBatch(long size)

Returns a new BigtableIO.Write with the max bytes a batch can have. After this many bytes are accumulated, the elements will be wrapped up in a batch and sent to Bigtable.

Does not modify this object.

withMaxOutstandingElements

public BigtableIO.Write withMaxOutstandingElements(long count)

Returns a new BigtableIO.Write with the max number of outstanding elements allowed before enforcing flow control.

Does not modify this object.

withMaxOutstandingBytes

public BigtableIO.Write withMaxOutstandingBytes(long bytes)

Returns a new BigtableIO.Write with the max number of outstanding bytes allowed before enforcing flow control.

Does not modify this object.

withFlowControl

public BigtableIO.Write withFlowControl(boolean enableFlowControl)

Returns a new BigtableIO.Write with flow control enabled if enableFlowControl is true.

When enabled, traffic to Bigtable is automatically rate-limited to prevent overloading Bigtable clusters while keeping enough load to trigger Bigtable Autoscaling (if enabled) to provision more nodes as needed. It is different from the flow control set by withMaxOutstandingElements(long) and withMaxOutstandingBytes(long), which is always enabled on batch writes and limits the number of outstanding requests to the Bigtable server.

Does not modify this object.

withThrottlingTargetMs

@Deprecated
public BigtableIO.Write withThrottlingTargetMs(int throttlingTargetMs)

Deprecated. This method has been deprecated in Beam 2.60.0. It does not have an effect.

withThrottlingReportTargetMs

@Deprecated
public BigtableIO.Write withThrottlingReportTargetMs(int throttlingReportTargetMs)

Deprecated. This method has been deprecated in Beam 2.60.0. It does not have an effect.

withErrorHandler

public BigtableIO.Write withErrorHandler(ErrorHandler<BadRecord,?> badRecordErrorHandler)

withWriteResults

public BigtableIO.WriteWithResults withWriteResults()

Returns a BigtableIO.WriteWithResults that will emit a BigtableWriteResult for each batch of rows written.

expand

public PDone expand(PCollection<KV<com.google.protobuf.ByteString,java.lang.Iterable<com.google.bigtable.v2.Mutation>>> input)

Description copied from class: PTransform

Override this method to specify how this PTransform should be expanded on the given InputT.

NOTE: This method should not be called directly. Instead apply the PTransform should be applied to the InputT using the apply method.

Composite transforms, which are defined in terms of other transforms, should return the output of one of the composed transforms. Non-composite transforms, which do not apply any transforms internally, should return a new unbound output and register evaluators (via backend-specific registration methods).

Specified by:

expand in class PTransform<PCollection<KV<com.google.protobuf.ByteString,java.lang.Iterable<com.google.bigtable.v2.Mutation>>>,PDone>

validate

public void validate(PipelineOptions options)

Description copied from class: PTransform

Called before running the Pipeline to verify this transform is fully and correctly specified.

By default, does nothing.

Overrides:

validate in class PTransform<PCollection<KV<com.google.protobuf.ByteString,java.lang.Iterable<com.google.bigtable.v2.Mutation>>>,PDone>

populateDisplayData

public void populateDisplayData(DisplayData.Builder builder)

Description copied from class: PTransform

Register display data for the given transform or component.

populateDisplayData(DisplayData.Builder) is invoked by Pipeline runners to collect display data via DisplayData.from(HasDisplayData). Implementations may call super.populateDisplayData(builder) in order to register display data in the current namespace, but should otherwise use subcomponent.populateDisplayData(builder) to use the namespace of the subcomponent.

By default, does not register any display data. Implementors may override this method to provide their own display data.

Specified by:

populateDisplayData in interface HasDisplayData

Overrides:

populateDisplayData in class PTransform<PCollection<KV<com.google.protobuf.ByteString,java.lang.Iterable<com.google.bigtable.v2.Mutation>>>,PDone>

Parameters:

builder - The builder to populate with display data.

See Also:

HasDisplayData

toString

public final java.lang.String toString()

Overrides:

toString in class PTransform<PCollection<KV<com.google.protobuf.ByteString,java.lang.Iterable<com.google.bigtable.v2.Mutation>>>,PDone>