Package org.apache.beam.sdk.io

Interface UnboundedSource.CheckpointMark

All Known Implementing Classes:

CheckpointMarkImpl, CountingSource.CounterMark, EmptyCheckpointMark, KafkaCheckpointMark, SolaceCheckpointMark, UnboundedSource.CheckpointMark.NoopCheckpointMark

Enclosing class:

UnboundedSource<OutputT,CheckpointMarkT extends UnboundedSource.CheckpointMark>

public static interface UnboundedSource.CheckpointMark

A marker representing the progress and state of an UnboundedSource.UnboundedReader.

For example, this could be offsets in a set of files being read.

Note that: The implementations of this interface should be encodable (have an associated Coder).

Nested Class Summary

Nested Classes

Modifier and Type	Interface	Description
static final class	UnboundedSource.CheckpointMark.NoopCheckpointMark	A checkpoint mark that does nothing when finalized.

Field Summary

Fields

Modifier and Type	Field	Description
static final UnboundedSource.CheckpointMark.NoopCheckpointMark	NOOP_CHECKPOINT_MARK

Method Summary

Modifier and Type	Method	Description
void	finalizeCheckpoint()	Called by the system to signal that this checkpoint mark has been committed along with all the records which have been read from the UnboundedSource.UnboundedReader since the previous checkpoint was taken.
default byte[]	getOffsetLimit()

Field Details

NOOP_CHECKPOINT_MARK

static final UnboundedSource.CheckpointMark.NoopCheckpointMark NOOP_CHECKPOINT_MARK

Method Details

finalizeCheckpoint

void finalizeCheckpoint()
                 throws IOException

Called by the system to signal that this checkpoint mark has been committed along with all the records which have been read from the UnboundedSource.UnboundedReader since the previous checkpoint was taken.

For example, this method could send acknowledgements to an external data source such as Pubsub.

Note that:

• This finalize method may be called from any thread, concurrently with calls to the UnboundedSource.UnboundedReader it was created from.
• Checkpoints will not necessarily be finalized as soon as they are created. A checkpoint may be taken while a previous checkpoint from the same UnboundedSource.UnboundedReader has not yet be finalized.
• In the absence of failures, all checkpoints will be finalized and they will be finalized in the same order they were taken from the UnboundedSource.UnboundedReader.
• It is possible for a checkpoint to be taken but this method never called. This method will never be called if the checkpoint could not be committed, and other failures may cause this method to never be called.
• It is not safe to assume the UnboundedSource.UnboundedReader from which this checkpoint was created still exists at the time this method is called.

Throws:

IOException

getOffsetLimit

default byte[] getOffsetLimit()