Package org.apache.beam.sdk.io.gcp.spanner.changestreams.action

Class PartitionStartRecordAction

java.lang.Object

org.apache.beam.sdk.io.gcp.spanner.changestreams.action.PartitionStartRecordAction

public class PartitionStartRecordAction
extends Object

This class is part of the process for invalid input: '{@link org.apache.beam.sdk.io.gcp.spanner.changestreams.dofn..ReadChangeStreamPartitionDoFn'} SDF. It is responsible for processing PartitionStartRecords. The new partition start records will be stored in the Connector's metadata tables in order to be scheduled for future querying by the

invalid reference

org.apache.beam.sdk.io.gcp.spanner.changestreams.dofn.dofn.DetectNewPartitionsDoFn

SDF.

Method Summary

Modifier and Type	Method	Description
Optional<DoFn.ProcessContinuation>	run(PartitionMetadata partition, PartitionStartRecord record, RestrictionTracker<TimestampRange,com.google.cloud.Timestamp> tracker, RestrictionInterrupter<com.google.cloud.Timestamp> interrupter, ManualWatermarkEstimator<Instant> watermarkEstimator)	This is the main processing function for a PartitionStartRecord.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Details

run

public Optional<DoFn.ProcessContinuation> run(PartitionMetadata partition,
 PartitionStartRecord record,
 RestrictionTracker<TimestampRange,com.google.cloud.Timestamp> tracker,
 RestrictionInterrupter<com.google.cloud.Timestamp> interrupter,
 ManualWatermarkEstimator<Instant> watermarkEstimator)

This is the main processing function for a PartitionStartRecord. It returns an Optional of DoFn.ProcessContinuation to indicate if the calling function should stop or not. If the Optional returned is empty, it means that the calling function can continue with the processing. If an Optional of DoFn.ProcessContinuation.stop() is returned, it means that this function was unable to claim the timestamp of the PartitionStartRecord, so the caller should stop.

When processing the PartitionStartRecord the following procedure is applied:

1. We try to claim the partition start record timestamp. If it is not possible, we stop here and return.
2. We update the watermark to the partition start record timestamp.
3. For each partition start record, we try to insert them in the metadata tables if they do not exist.
4. For each partition start record, we increment the corresponding metric.

Parameters:

partition - the current partition being processed

record - the change stream partition start record received

tracker - the restriction tracker of the

invalid reference

com.google.cloud.teleport.spanner.spannerio.changestreams.dofn.ReadChangeStreamPartitionDoFn

SDF

interrupter - the restriction interrupter suggesting early termination of the processing

watermarkEstimator - the watermark estimator of the

invalid reference

com.google.cloud.teleport.spanner.spannerio.changestreams.dofn.ReadChangeStreamPartitionDoFn

SDF

Returns:

Optional.empty() if the caller can continue processing more records. A non empty Optional with DoFn.ProcessContinuation.stop() if this function was unable to claim the

invalid reference

ChildPartitionsRecord

timestamp. A non empty Optional with DoFn.ProcessContinuation.resume() if this function should commit what has already been processed and resume.