org.apache.beam.sdk.io.gcp.bigtable.changestreams.action

Class DetectNewPartitionsAction

java.lang.Object

org.apache.beam.sdk.io.gcp.bigtable.changestreams.action.DetectNewPartitionsAction

@Internal
public class DetectNewPartitionsAction
extends java.lang.Object

This class processes DetectNewPartitionsDoFn.

Constructor Summary

Constructors

Constructor and Description
DetectNewPartitionsAction(ChangeStreamMetrics metrics, MetadataTableDao metadataTableDao, GenerateInitialPartitionsAction generateInitialPartitionsAction)

Method Summary

Modifier and Type	Method and Description
DoFn.ProcessContinuation	run(RestrictionTracker<OffsetRange,java.lang.Long> tracker, DoFn.OutputReceiver<PartitionRecord> receiver, ManualWatermarkEstimator<Instant> watermarkEstimator, DoFn.BundleFinalizer bundleFinalizer, Instant startTime) Perform the necessary steps to manage initial set of partitions and new partitions.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

DetectNewPartitionsAction

public DetectNewPartitionsAction(ChangeStreamMetrics metrics,
                                 MetadataTableDao metadataTableDao,
                                 GenerateInitialPartitionsAction generateInitialPartitionsAction)

Method Detail

run

public DoFn.ProcessContinuation run(RestrictionTracker<OffsetRange,java.lang.Long> tracker,
                                    DoFn.OutputReceiver<PartitionRecord> receiver,
                                    ManualWatermarkEstimator<Instant> watermarkEstimator,
                                    DoFn.BundleFinalizer bundleFinalizer,
                                    Instant startTime)
                             throws java.lang.Exception

Perform the necessary steps to manage initial set of partitions and new partitions. Currently, we set to process new partitions every second.

1. Look up the initial list of partitions to stream if it's the very first run.
2. On rest of the runs, try advancing watermark if needed.
3. Update the metadata table with info about this DoFn.
4. Process new partitions and output them.
5. Reconcile any Partitions that haven't been streaming for a long time
6. Register callback to clean up processed partitions after bundle has been finalized.

Parameters:

tracker - offset tracker that simply increment by 1 every single run

receiver - output new partitions

watermarkEstimator - update watermark that is a representation of the low watermark of the entire beam pipeline

bundleFinalizer - perform after bundle output actions to clean up metadata table

Returns:

DoFn.ProcessContinuation.resume() with 1-second delay if the stream continues, otherwise DoFn.ProcessContinuation.stop()

Throws:

com.google.protobuf.InvalidProtocolBufferException - if failing to process new partitions

java.lang.Exception