Package org.apache.beam.sdk.io.kafka

Class KafkaCheckpointMark

java.lang.Object

org.apache.beam.sdk.io.kafka.KafkaCheckpointMark

All Implemented Interfaces:

UnboundedSource.CheckpointMark

@DefaultCoder(AvroCoder.class)
public class KafkaCheckpointMark
extends Object
implements UnboundedSource.CheckpointMark

Checkpoint for a KafkaUnboundedReader. Consists of Kafka topic name, partition id, and the latest offset consumed so far.

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static class	KafkaCheckpointMark.PartitionMark	A tuple to hold topic, partition, and offset that comprise the checkpoint for a single partition.

Nested classes/interfaces inherited from interface org.apache.beam.sdk.io.UnboundedSource.CheckpointMark

UnboundedSource.CheckpointMark.NoopCheckpointMark

Field Summary

Fields inherited from interface org.apache.beam.sdk.io.UnboundedSource.CheckpointMark

NOOP_CHECKPOINT_MARK

Constructor Summary

Constructors

Constructor	Description
KafkaCheckpointMark(List<KafkaCheckpointMark.PartitionMark> partitions, Optional<org.apache.beam.sdk.io.kafka.KafkaUnboundedReader<?,?>> reader)

Method Summary

Modifier and Type	Method	Description
void	finalizeCheckpoint()	Called by the system to signal that this checkpoint mark has been committed along with all the records which have been read from the UnboundedSource.UnboundedReader since the previous checkpoint was taken.
byte[]	getOffsetLimit()
List<KafkaCheckpointMark.PartitionMark>	getPartitions()
String	toString()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Details

KafkaCheckpointMark

public KafkaCheckpointMark(List<KafkaCheckpointMark.PartitionMark> partitions,
 Optional<org.apache.beam.sdk.io.kafka.KafkaUnboundedReader<?,?>> reader)

Method Details

getPartitions

public List<KafkaCheckpointMark.PartitionMark> getPartitions()

finalizeCheckpoint

public void finalizeCheckpoint()

Description copied from interface: UnboundedSource.CheckpointMark

Called by the system to signal that this checkpoint mark has been committed along with all the records which have been read from the UnboundedSource.UnboundedReader since the previous checkpoint was taken.

For example, this method could send acknowledgements to an external data source such as Pubsub.

Note that:

• This finalize method may be called from any thread, concurrently with calls to the UnboundedSource.UnboundedReader it was created from.
• Checkpoints will not necessarily be finalized as soon as they are created. A checkpoint may be taken while a previous checkpoint from the same UnboundedSource.UnboundedReader has not yet be finalized.
• In the absence of failures, all checkpoints will be finalized and they will be finalized in the same order they were taken from the UnboundedSource.UnboundedReader.
• It is possible for a checkpoint to be taken but this method never called. This method will never be called if the checkpoint could not be committed, and other failures may cause this method to never be called.
• It is not safe to assume the UnboundedSource.UnboundedReader from which this checkpoint was created still exists at the time this method is called.

Specified by:

finalizeCheckpoint in interface UnboundedSource.CheckpointMark

toString

public String toString()

Overrides:

toString in class Object

getOffsetLimit

public byte[] getOffsetLimit()

Specified by:

getOffsetLimit in interface UnboundedSource.CheckpointMark