Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
public DoFn.ProcessContinuation run(RestrictionTracker<OffsetRange,java.lang.Long> tracker,
Perform the necessary steps to manage initial set of partitions and new partitions. Currently,
we set to process new partitions every second.
- Look up the initial list of partitions to stream if it's the very first run.
- On rest of the runs, try advancing watermark if needed.
- Update the metadata table with info about this DoFn.
- Check if this pipeline has reached the end time. Terminate if it has.
- Process new partitions and output them.
- Reconcile any Partitions that haven't been streaming for a long time
- Register callback to clean up processed partitions after bundle has been finalized.
tracker - offset tracker that simply increment by 1 every single run
receiver - output new partitions
watermarkEstimator - update watermark that is a representation of the low watermark of the
entire beam pipeline
DoFn.ProcessContinuation.resume() with 1-second delay if the stream continues,
com.google.protobuf.InvalidProtocolBufferException - if failing to process new partitions