Package org.apache.beam.sdk.io.gcp.bigquery

Class SchemaUpdateHoldingFn<DestinationT extends @NonNull Object,ElementT>

java.lang.Object

org.apache.beam.sdk.transforms.DoFn<KV<org.apache.beam.sdk.util.ShardedKey<DestinationT>,@Nullable ElementT>,KV<DestinationT,StorageApiWritePayload>>

org.apache.beam.sdk.io.gcp.bigquery.SchemaUpdateHoldingFn<DestinationT,ElementT>

All Implemented Interfaces:

Serializable, HasDisplayData

public class SchemaUpdateHoldingFn<DestinationT extends @NonNull Object,ElementT>
extends DoFn<KV<org.apache.beam.sdk.util.ShardedKey<DestinationT>,@Nullable ElementT>,KV<DestinationT,StorageApiWritePayload>>

This is a stateful DoFn that buffers elements that triggered table schema update. Once the table schema has been updated, this reprocesses the messages and allows them to continue on through the sink. This DoFn receives messages from both ConvertMessagesDoFn and PatchTableSchemaDoFn. ConvertMessagesDoFn sends elements to be buffered. PatchTableSchemaDoFn sends a null element as a sentinal to indicate that the table has recently been patched, which triggers us to immediately try and reprocess the buffered elements.

See Also:

• Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.beam.sdk.transforms.DoFn

DoFn.AlwaysFetched, DoFn.BoundedPerElement, DoFn.BundleFinalizer, DoFn.CurrentRecordId, DoFn.CurrentRecordOffset, DoFn.Element, DoFn.FieldAccess, DoFn.FinishBundle, DoFn.FinishBundleContext, DoFn.FireTimestamp, DoFn.GetInitialRestriction, DoFn.GetInitialWatermarkEstimatorState, DoFn.GetRestrictionCoder, DoFn.GetSize, DoFn.GetWatermarkEstimatorStateCoder, DoFn.Key, DoFn.MultiOutputReceiver, DoFn.NewTracker, DoFn.NewWatermarkEstimator, DoFn.OnTimer, DoFn.OnTimerContext, DoFn.OnTimerFamily, DoFn.OnWindowExpiration, DoFn.OnWindowExpirationContext, DoFn.OutputReceiver<T>, DoFn.ProcessContext, DoFn.ProcessContinuation, DoFn.ProcessElement, DoFn.RequiresStableInput, DoFn.RequiresTimeSortedInput, DoFn.Restriction, DoFn.Setup, DoFn.SideInput, DoFn.SplitRestriction, DoFn.StartBundle, DoFn.StartBundleContext, DoFn.StateId, DoFn.Teardown, DoFn.TimerFamily, DoFn.TimerId, DoFn.Timestamp, DoFn.TruncateRestriction, DoFn.UnboundedPerElement, DoFn.WatermarkEstimatorState, DoFn.WindowedContext

Constructor Summary

Constructors

Constructor	Description
SchemaUpdateHoldingFn(Coder<ElementT> elementCoder, ConvertMessagesDoFn<DestinationT,ElementT> convertMessagesDoFn)

Method Summary

Modifier and Type	Method	Description
Duration	getAllowedTimestampSkew()	Returns the allowed timestamp skew duration, which is the maximum duration that timestamps can be shifted backward in DoFn.WindowedContext.outputWithTimestamp(OutputT, org.joda.time.Instant).
void	onPollTimer(org.apache.beam.sdk.util.ShardedKey<DestinationT> key, PipelineOptions pipelineOptions, BagState<TimestampedValue<ElementT>> bag, CombiningState<Long,long[],Long> minBufferedTimestamp, ValueState<Long> timerTs, Timer pollTimer, BoundedWindow window, DoFn.MultiOutputReceiver o)
void	onTeardown()
void	onWindowExpiration(org.apache.beam.sdk.util.ShardedKey<DestinationT> key, PipelineOptions pipelineOptions, BagState<TimestampedValue<ElementT>> bag, CombiningState<Long,long[],Long> minBufferedTimestamp, DoFn.MultiOutputReceiver o)
void	processElement(KV<org.apache.beam.sdk.util.ShardedKey<DestinationT>,@Nullable ElementT> element, Instant timestamp, BagState<TimestampedValue<ElementT>> bag, CombiningState<Long,long[],Long> minBufferedTimestamp, ValueState<Long> timerTs, Timer pollTimer, DoFn<KV<org.apache.beam.sdk.util.ShardedKey<DestinationT>,@Nullable ElementT>,KV<DestinationT,StorageApiWritePayload>>.ProcessContext context, BoundedWindow window, DoFn.MultiOutputReceiver o)
void	startBundle()
boolean	tryFlushBuffer(DestinationT destination, PipelineOptions pipelineOptions, BagState<TimestampedValue<ElementT>> bag, CombiningState<Long,long[],Long> minBufferedTimestamp, DoFn.MultiOutputReceiver o)

Methods inherited from class org.apache.beam.sdk.transforms.DoFn

getInputTypeDescriptor, getOutputTypeDescriptor, populateDisplayData, prepareForProcessing

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

SchemaUpdateHoldingFn

public SchemaUpdateHoldingFn(Coder<ElementT> elementCoder,
 ConvertMessagesDoFn<DestinationT,ElementT> convertMessagesDoFn)

Method Details

startBundle

@StartBundle
public void startBundle()

onTeardown

@Teardown
public void onTeardown()

processElement

@ProcessElement
public void processElement(@Element
 KV<org.apache.beam.sdk.util.ShardedKey<DestinationT>,@Nullable ElementT> element,
 @Timestamp
 Instant timestamp,
 @StateId("bufferedElements")
 BagState<TimestampedValue<ElementT>> bag,
 @StateId("minBufferedTimestamp")
 CombiningState<Long,long[],Long> minBufferedTimestamp,
 @StateId("timerTimestamp")
 ValueState<Long> timerTs,
 @TimerId("pollTimer")
 Timer pollTimer,
 DoFn<KV<org.apache.beam.sdk.util.ShardedKey<DestinationT>,@Nullable ElementT>,KV<DestinationT,StorageApiWritePayload>>.ProcessContext context,
 BoundedWindow window,
 DoFn.MultiOutputReceiver o)
                    throws Exception

Throws:

Exception

getAllowedTimestampSkew

public Duration getAllowedTimestampSkew()

Description copied from class: DoFn

Returns the allowed timestamp skew duration, which is the maximum duration that timestamps can be shifted backward in DoFn.WindowedContext.outputWithTimestamp(OutputT, org.joda.time.Instant).

The default value is Duration.ZERO, in which case timestamps can only be shifted forward to future. For infinite skew, return Duration.millis(Long.MAX_VALUE).

Overrides:

getAllowedTimestampSkew in class DoFn<KV<org.apache.beam.sdk.util.ShardedKey<DestinationT extends @NonNull Object>,@Nullable ElementT>,KV<DestinationT extends @NonNull Object,StorageApiWritePayload>>

onPollTimer

@OnTimer("pollTimer")
public void onPollTimer(@Key
 org.apache.beam.sdk.util.ShardedKey<DestinationT> key,
 PipelineOptions pipelineOptions,
 @StateId("bufferedElements")
 BagState<TimestampedValue<ElementT>> bag,
 @StateId("minBufferedTimestamp")
 CombiningState<Long,long[],Long> minBufferedTimestamp,
 @StateId("timerTimestamp")
 ValueState<Long> timerTs,
 @TimerId("pollTimer")
 Timer pollTimer,
 BoundedWindow window,
 DoFn.MultiOutputReceiver o)
                 throws Exception

Throws:

Exception

onWindowExpiration

@OnWindowExpiration
public void onWindowExpiration(@Key
 org.apache.beam.sdk.util.ShardedKey<DestinationT> key,
 PipelineOptions pipelineOptions,
 @StateId("bufferedElements")
 BagState<TimestampedValue<ElementT>> bag,
 @StateId("minBufferedTimestamp")
 CombiningState<Long,long[],Long> minBufferedTimestamp,
 DoFn.MultiOutputReceiver o)
                        throws Exception

Throws:

Exception

tryFlushBuffer

public boolean tryFlushBuffer(DestinationT destination,
 PipelineOptions pipelineOptions,
 @StateId("bufferedElements")
 BagState<TimestampedValue<ElementT>> bag,
 @StateId("minBufferedTimestamp")
 CombiningState<Long,long[],Long> minBufferedTimestamp,
 DoFn.MultiOutputReceiver o)
                       throws Exception

Throws:

Exception