java.lang.Object
org.apache.beam.sdk.transforms.DoFn<KV<org.apache.beam.sdk.util.ShardedKey<DestinationT>,@Nullable ElementT>,KV<DestinationT,StorageApiWritePayload>>
org.apache.beam.sdk.io.gcp.bigquery.SchemaUpdateHoldingFn<DestinationT,ElementT>
- All Implemented Interfaces:
Serializable,HasDisplayData
public class SchemaUpdateHoldingFn<DestinationT extends @NonNull Object,ElementT>
extends DoFn<KV<org.apache.beam.sdk.util.ShardedKey<DestinationT>,@Nullable ElementT>,KV<DestinationT,StorageApiWritePayload>>
This is a stateful DoFn that buffers elements that triggered table schema update. Once the table
schema has been updated, this reprocesses the messages and allows them to continue on through the
sink. This DoFn receives messages from both
ConvertMessagesDoFn and PatchTableSchemaDoFn. ConvertMessagesDoFn sends elements to be buffered. PatchTableSchemaDoFn sends a null element as a sentinal to indicate that the table has recently
been patched, which triggers us to immediately try and reprocess the buffered elements.- See Also:
-
Nested Class Summary
Nested classes/interfaces inherited from class org.apache.beam.sdk.transforms.DoFn
DoFn.AlwaysFetched, DoFn.BoundedPerElement, DoFn.BundleFinalizer, DoFn.CurrentRecordId, DoFn.CurrentRecordOffset, DoFn.Element, DoFn.FieldAccess, DoFn.FinishBundle, DoFn.FinishBundleContext, DoFn.FireTimestamp, DoFn.GetInitialRestriction, DoFn.GetInitialWatermarkEstimatorState, DoFn.GetRestrictionCoder, DoFn.GetSize, DoFn.GetWatermarkEstimatorStateCoder, DoFn.Key, DoFn.MultiOutputReceiver, DoFn.NewTracker, DoFn.NewWatermarkEstimator, DoFn.OnTimer, DoFn.OnTimerContext, DoFn.OnTimerFamily, DoFn.OnWindowExpiration, DoFn.OnWindowExpirationContext, DoFn.OutputReceiver<T>, DoFn.ProcessContext, DoFn.ProcessContinuation, DoFn.ProcessElement, DoFn.RequiresStableInput, DoFn.RequiresTimeSortedInput, DoFn.Restriction, DoFn.Setup, DoFn.SideInput, DoFn.SplitRestriction, DoFn.StartBundle, DoFn.StartBundleContext, DoFn.StateId, DoFn.Teardown, DoFn.TimerFamily, DoFn.TimerId, DoFn.Timestamp, DoFn.TruncateRestriction, DoFn.UnboundedPerElement, DoFn.WatermarkEstimatorState, DoFn.WindowedContext -
Constructor Summary
ConstructorsConstructorDescriptionSchemaUpdateHoldingFn(Coder<ElementT> elementCoder, ConvertMessagesDoFn<DestinationT, ElementT> convertMessagesDoFn) -
Method Summary
Modifier and TypeMethodDescriptionReturns the allowed timestamp skew duration, which is the maximum duration that timestamps can be shifted backward inDoFn.WindowedContext.outputWithTimestamp(OutputT, org.joda.time.Instant).voidonPollTimer(DoFn<KV<org.apache.beam.sdk.util.ShardedKey<DestinationT>, @Nullable ElementT>, KV<DestinationT, StorageApiWritePayload>>.OnTimerContext context, org.apache.beam.sdk.util.ShardedKey<DestinationT> key, PipelineOptions pipelineOptions, BagState<TimestampedValue<ElementT>> bag, CombiningState<Long, long[], Long> minBufferedTimestamp, ValueState<Long> timerTs, Timer pollTimer, BoundedWindow window, DoFn.MultiOutputReceiver o) voidvoidonWindowExpiration(DoFn<KV<org.apache.beam.sdk.util.ShardedKey<DestinationT>, @Nullable ElementT>, KV<DestinationT, StorageApiWritePayload>>.OnWindowExpirationContext context, org.apache.beam.sdk.util.ShardedKey<DestinationT> key, PipelineOptions pipelineOptions, BagState<TimestampedValue<ElementT>> bag, CombiningState<Long, long[], Long> minBufferedTimestamp, BoundedWindow window, DoFn.MultiOutputReceiver o) voidprocessElement(KV<org.apache.beam.sdk.util.ShardedKey<DestinationT>, @Nullable ElementT> element, Instant timestamp, BagState<TimestampedValue<ElementT>> bag, CombiningState<Long, long[], Long> minBufferedTimestamp, ValueState<Long> timerTs, Timer pollTimer, DoFn<KV<org.apache.beam.sdk.util.ShardedKey<DestinationT>, @Nullable ElementT>, KV<DestinationT, StorageApiWritePayload>>.ProcessContext context, BoundedWindow window, DoFn.MultiOutputReceiver o) voidbooleantryFlushBuffer(DestinationT destination, PipelineOptions pipelineOptions, BagState<TimestampedValue<ElementT>> bag, CombiningState<Long, long[], Long> minBufferedTimestamp, DoFn.MultiOutputReceiver o, boolean collectSchemaErrors) Methods inherited from class org.apache.beam.sdk.transforms.DoFn
getInputTypeDescriptor, getOutputTypeDescriptor, populateDisplayData, prepareForProcessing
-
Constructor Details
-
SchemaUpdateHoldingFn
public SchemaUpdateHoldingFn(Coder<ElementT> elementCoder, ConvertMessagesDoFn<DestinationT, ElementT> convertMessagesDoFn)
-
-
Method Details
-
startBundle
-
onTeardown
-
processElement
@ProcessElement public void processElement(@Element KV<org.apache.beam.sdk.util.ShardedKey<DestinationT>, @Nullable ElementT> element, @Timestamp Instant timestamp, @StateId("bufferedElements") BagState<TimestampedValue<ElementT>> bag, @StateId("minBufferedTimestamp") CombiningState<Long, throws Exceptionlong[], Long> minBufferedTimestamp, @StateId("timerTimestamp") ValueState<Long> timerTs, @TimerId("pollTimer") Timer pollTimer, DoFn<KV<org.apache.beam.sdk.util.ShardedKey<DestinationT>, @Nullable ElementT>, KV<DestinationT, StorageApiWritePayload>>.ProcessContext context, BoundedWindow window, DoFn.MultiOutputReceiver o) - Throws:
Exception
-
getAllowedTimestampSkew
Description copied from class:DoFnReturns the allowed timestamp skew duration, which is the maximum duration that timestamps can be shifted backward inDoFn.WindowedContext.outputWithTimestamp(OutputT, org.joda.time.Instant).The default value is
Duration.ZERO, in which case timestamps can only be shifted forward to future. For infinite skew, returnDuration.millis(Long.MAX_VALUE).- Overrides:
getAllowedTimestampSkewin classDoFn<KV<org.apache.beam.sdk.util.ShardedKey<DestinationT extends @NonNull Object>,@Nullable ElementT>, KV<DestinationT extends @NonNull Object, StorageApiWritePayload>>
-
onPollTimer
@OnTimer("pollTimer") public void onPollTimer(DoFn<KV<org.apache.beam.sdk.util.ShardedKey<DestinationT>, @Nullable ElementT>, throws ExceptionKV<DestinationT, StorageApiWritePayload>>.OnTimerContext context, @Key org.apache.beam.sdk.util.ShardedKey<DestinationT> key, PipelineOptions pipelineOptions, @StateId("bufferedElements") BagState<TimestampedValue<ElementT>> bag, @StateId("minBufferedTimestamp") CombiningState<Long, long[], Long> minBufferedTimestamp, @StateId("timerTimestamp") ValueState<Long> timerTs, @TimerId("pollTimer") Timer pollTimer, BoundedWindow window, DoFn.MultiOutputReceiver o) - Throws:
Exception
-
onWindowExpiration
@OnWindowExpiration public void onWindowExpiration(DoFn<KV<org.apache.beam.sdk.util.ShardedKey<DestinationT>, @Nullable ElementT>, throws ExceptionKV<DestinationT, StorageApiWritePayload>>.OnWindowExpirationContext context, @Key org.apache.beam.sdk.util.ShardedKey<DestinationT> key, PipelineOptions pipelineOptions, @StateId("bufferedElements") BagState<TimestampedValue<ElementT>> bag, @StateId("minBufferedTimestamp") CombiningState<Long, long[], Long> minBufferedTimestamp, BoundedWindow window, DoFn.MultiOutputReceiver o) - Throws:
Exception
-
tryFlushBuffer
public boolean tryFlushBuffer(DestinationT destination, PipelineOptions pipelineOptions, @StateId("bufferedElements") BagState<TimestampedValue<ElementT>> bag, @StateId("minBufferedTimestamp") CombiningState<Long, long[], throws ExceptionLong> minBufferedTimestamp, DoFn.MultiOutputReceiver o, boolean collectSchemaErrors) - Throws:
Exception
-