org.apache.beam.sdk.io.gcp.bigtable.changestreams.dofn

Class ReadChangeStreamPartitionDoFn

java.lang.Object

org.apache.beam.sdk.transforms.DoFn<PartitionRecord,KV<com.google.protobuf.ByteString,com.google.cloud.bigtable.data.v2.models.ChangeStreamMutation>>

org.apache.beam.sdk.io.gcp.bigtable.changestreams.dofn.ReadChangeStreamPartitionDoFn

All Implemented Interfaces:

java.io.Serializable, HasDisplayData

@Internal
 @DoFn.UnboundedPerElement
public class ReadChangeStreamPartitionDoFn
extends DoFn<PartitionRecord,KV<com.google.protobuf.ByteString,com.google.cloud.bigtable.data.v2.models.ChangeStreamMutation>>

See Also:

Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.beam.sdk.transforms.DoFn

DoFn.AlwaysFetched, DoFn.BoundedPerElement, DoFn.BundleFinalizer, DoFn.Element, DoFn.FieldAccess, DoFn.FinishBundle, DoFn.FinishBundleContext, DoFn.GetInitialRestriction, DoFn.GetInitialWatermarkEstimatorState, DoFn.GetRestrictionCoder, DoFn.GetSize, DoFn.GetWatermarkEstimatorStateCoder, DoFn.Key, DoFn.MultiOutputReceiver, DoFn.NewTracker, DoFn.NewWatermarkEstimator, DoFn.OnTimer, DoFn.OnTimerContext, DoFn.OnTimerFamily, DoFn.OnWindowExpiration, DoFn.OnWindowExpirationContext, DoFn.OutputReceiver<T>, DoFn.ProcessContext, DoFn.ProcessContinuation, DoFn.ProcessElement, DoFn.RequiresStableInput, DoFn.RequiresTimeSortedInput, DoFn.Restriction, DoFn.Setup, DoFn.SideInput, DoFn.SplitRestriction, DoFn.StartBundle, DoFn.StartBundleContext, DoFn.StateId, DoFn.Teardown, DoFn.TimerFamily, DoFn.TimerId, DoFn.Timestamp, DoFn.TruncateRestriction, DoFn.UnboundedPerElement, DoFn.WatermarkEstimatorState, DoFn.WindowedContext

Field Summary

Fields

Modifier and Type	Field and Description
static int	THROUGHPUT_ESTIMATION_WINDOW_SECONDS

Constructor Summary

Constructors

Constructor and Description
ReadChangeStreamPartitionDoFn(Duration heartbeatDuration, DaoFactory daoFactory, ActionFactory actionFactory, ChangeStreamMetrics metrics)

Method Summary

Modifier and Type	Method and Description
Instant	getInitialWatermarkEstimatorState(PartitionRecord partitionRecord)
double	getSize(StreamProgress streamProgress)
StreamProgress	initialRestriction()
ManualWatermarkEstimator<Instant>	newWatermarkEstimator(Instant watermarkEstimatorState)
DoFn.ProcessContinuation	processElement(PartitionRecord partitionRecord, RestrictionTracker<StreamProgress,StreamProgress> tracker, DoFn.OutputReceiver<KV<com.google.protobuf.ByteString,com.google.cloud.bigtable.data.v2.models.ChangeStreamMutation>> receiver, ManualWatermarkEstimator<Instant> watermarkEstimator)
ReadChangeStreamPartitionProgressTracker	restrictionTracker(StreamProgress restriction)
void	setThroughputEstimator(BytesThroughputEstimator<KV<com.google.protobuf.ByteString,com.google.cloud.bigtable.data.v2.models.ChangeStreamMutation>> throughputEstimator) Sets the estimator to track throughput for each DoFn instance.
void	setup()

Methods inherited from class org.apache.beam.sdk.transforms.DoFn

getAllowedTimestampSkew, getInputTypeDescriptor, getOutputTypeDescriptor, populateDisplayData, prepareForProcessing

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

THROUGHPUT_ESTIMATION_WINDOW_SECONDS

public static final int THROUGHPUT_ESTIMATION_WINDOW_SECONDS

See Also:

Constant Field Values

Constructor Detail

ReadChangeStreamPartitionDoFn

public ReadChangeStreamPartitionDoFn(Duration heartbeatDuration,
                                     DaoFactory daoFactory,
                                     ActionFactory actionFactory,
                                     ChangeStreamMetrics metrics)

Method Detail

getInitialWatermarkEstimatorState

@DoFn.GetInitialWatermarkEstimatorState
public Instant getInitialWatermarkEstimatorState(@DoFn.Element
                                                                                         PartitionRecord partitionRecord)

newWatermarkEstimator

@DoFn.NewWatermarkEstimator
public ManualWatermarkEstimator<Instant> newWatermarkEstimator(@DoFn.WatermarkEstimatorState
                                                                                           Instant watermarkEstimatorState)

initialRestriction

@DoFn.GetInitialRestriction
public StreamProgress initialRestriction()

restrictionTracker

@DoFn.NewTracker
public ReadChangeStreamPartitionProgressTracker restrictionTracker(@DoFn.Restriction
                                                                                    StreamProgress restriction)

getSize

@DoFn.GetSize
public double getSize(@DoFn.Restriction
                                    StreamProgress streamProgress)

setup

@DoFn.Setup
public void setup()
                       throws java.io.IOException

Throws:

java.io.IOException

processElement

@DoFn.ProcessElement
public DoFn.ProcessContinuation processElement(@DoFn.Element
                                                                    PartitionRecord partitionRecord,
                                                                    RestrictionTracker<StreamProgress,StreamProgress> tracker,
                                                                    DoFn.OutputReceiver<KV<com.google.protobuf.ByteString,com.google.cloud.bigtable.data.v2.models.ChangeStreamMutation>> receiver,
                                                                    ManualWatermarkEstimator<Instant> watermarkEstimator)
                                                             throws java.lang.InterruptedException,
                                                                    java.io.IOException

Throws:

java.lang.InterruptedException

java.io.IOException

setThroughputEstimator

public void setThroughputEstimator(BytesThroughputEstimator<KV<com.google.protobuf.ByteString,com.google.cloud.bigtable.data.v2.models.ChangeStreamMutation>> throughputEstimator)

Sets the estimator to track throughput for each DoFn instance.

Parameters:

throughputEstimator - an estimator to calculate DoFn instance level throughput