Package org.apache.beam.sdk.io.gcp.bigquery

Class ConvertMessagesDoFn<DestinationT extends @NonNull Object,ElementT>

java.lang.Object

org.apache.beam.sdk.transforms.DoFn<KV<DestinationT,ElementT>,KV<DestinationT,StorageApiWritePayload>>

org.apache.beam.sdk.io.gcp.bigquery.ConvertMessagesDoFn<DestinationT,ElementT>

All Implemented Interfaces:

Serializable, HasDisplayData

public class ConvertMessagesDoFn<DestinationT extends @NonNull Object,ElementT>
extends DoFn<KV<DestinationT,ElementT>,KV<DestinationT,StorageApiWritePayload>>

DoFn that interacts with the StorageApiDynamicDestinations instance to convert messages to StorageApiWritePaylod. Messages that fail to convert are routed to the dead-letter PCollection. If schemaUpdateOptions are set, then messages that fail to convert due to missing columns are routed to a buffering transform that holds them until the table's schema has been updated.

See Also:

• Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.beam.sdk.transforms.DoFn

DoFn.AlwaysFetched, DoFn.BoundedPerElement, DoFn.BundleFinalizer, DoFn.CurrentRecordId, DoFn.CurrentRecordOffset, DoFn.Element, DoFn.FieldAccess, DoFn.FinishBundle, DoFn.FinishBundleContext, DoFn.FireTimestamp, DoFn.GetInitialRestriction, DoFn.GetInitialWatermarkEstimatorState, DoFn.GetRestrictionCoder, DoFn.GetSize, DoFn.GetWatermarkEstimatorStateCoder, DoFn.Key, DoFn.MultiOutputReceiver, DoFn.NewTracker, DoFn.NewWatermarkEstimator, DoFn.OnTimer, DoFn.OnTimerContext, DoFn.OnTimerFamily, DoFn.OnWindowExpiration, DoFn.OnWindowExpirationContext, DoFn.OutputReceiver<T>, DoFn.ProcessContext, DoFn.ProcessContinuation, DoFn.ProcessElement, DoFn.RequiresStableInput, DoFn.RequiresTimeSortedInput, DoFn.Restriction, DoFn.Setup, DoFn.SideInput, DoFn.SplitRestriction, DoFn.StartBundle, DoFn.StartBundleContext, DoFn.StateId, DoFn.Teardown, DoFn.TimerFamily, DoFn.TimerId, DoFn.Timestamp, DoFn.TruncateRestriction, DoFn.UnboundedPerElement, DoFn.WatermarkEstimatorState, DoFn.WindowedContext

Method Summary

Modifier and Type	Method	Description
void	finishBundle(DoFn<KV<DestinationT,ElementT>,KV<DestinationT,StorageApiWritePayload>>.FinishBundleContext c)
Iterable<TimestampedValue<KV<DestinationT,ElementT>>>	handleProcessElements(org.apache.beam.sdk.io.gcp.bigquery.StorageApiDynamicDestinations.MessageConverter<ElementT> messageConverter, Iterable<TimestampedValue<KV<DestinationT,ElementT>>> values, DoFn.MultiOutputReceiver o, TableRowToStorageApiProto.ErrorCollector errorCollector)
void	onTeardown()
void	processElement(DoFn<KV<DestinationT,ElementT>,KV<DestinationT,StorageApiWritePayload>>.ProcessContext c, PipelineOptions pipelineOptions, KV<DestinationT,ElementT> element, Instant timestamp, DoFn.MultiOutputReceiver o)
void	startBundle()

Methods inherited from class org.apache.beam.sdk.transforms.DoFn

getAllowedTimestampSkew, getInputTypeDescriptor, getOutputTypeDescriptor, populateDisplayData, prepareForProcessing

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Details

onTeardown

@Teardown
public void onTeardown()

startBundle

@StartBundle
public void startBundle()

processElement

@ProcessElement
public void processElement(DoFn<KV<DestinationT,ElementT>,KV<DestinationT,StorageApiWritePayload>>.ProcessContext c,
 PipelineOptions pipelineOptions,
 @Element
 KV<DestinationT,ElementT> element,
 @Timestamp
 Instant timestamp,
 DoFn.MultiOutputReceiver o)
                    throws Exception

Throws:

Exception

finishBundle

@FinishBundle
public void finishBundle(DoFn<KV<DestinationT,ElementT>,KV<DestinationT,StorageApiWritePayload>>.FinishBundleContext c)
                  throws Exception

Throws:

Exception

handleProcessElements

public Iterable<TimestampedValue<KV<DestinationT,ElementT>>> handleProcessElements(org.apache.beam.sdk.io.gcp.bigquery.StorageApiDynamicDestinations.MessageConverter<ElementT> messageConverter,
 Iterable<TimestampedValue<KV<DestinationT,ElementT>>> values,
 DoFn.MultiOutputReceiver o,
 TableRowToStorageApiProto.ErrorCollector errorCollector)
                                                                            throws Exception

Throws:

Exception