org.apache.beam.sdk.io.kafka

Class TimestampPolicy<K,V>

java.lang.Object

org.apache.beam.sdk.io.kafka.TimestampPolicy<K,V>

Direct Known Subclasses:

CustomTimestampPolicyWithLimitedDelay, TimestampPolicyFactory.LogAppendTimePolicy, TimestampPolicyFactory.ProcessingTimePolicy, TimestampPolicyFactory.TimestampFnPolicy

public abstract class TimestampPolicy<K,V>
extends java.lang.Object

A timestamp policy to assign event time for messages in a Kafka partition and watermark for it. KafkaIO reader creates one policy using TimestampPolicyFactory for each each of the partitions it reads from. See @TimestampPolicyFactory.LogAppendTimePolicy for example of a policy.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	TimestampPolicy.PartitionContext The context contains state maintained in the reader for the partition.

Constructor Summary

Constructors

Constructor and Description
TimestampPolicy()

Method Summary

Modifier and Type	Method and Description
abstract Instant	getTimestampForRecord(TimestampPolicy.PartitionContext ctx, KafkaRecord<K,V> record) Returns record timestamp (aka event time).
abstract Instant	getWatermark(TimestampPolicy.PartitionContext ctx) Returns watermark for the partition.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

TimestampPolicy

public TimestampPolicy()

Method Detail

getTimestampForRecord

public abstract Instant getTimestampForRecord(TimestampPolicy.PartitionContext ctx,
                                              KafkaRecord<K,V> record)

Returns record timestamp (aka event time). This is often based on the timestamp of the Kafka record. This is invoked for each record when it is processed in the reader.

getWatermark

public abstract Instant getWatermark(TimestampPolicy.PartitionContext ctx)

Returns watermark for the partition. It is the timestamp before or at the timestamps of all future records consumed from the partition. See UnboundedSource.UnboundedReader#getWatermark() for more guidance on watermarks. E.g. if the record timestamp is 'LogAppendTime', watermark would be the timestamp of the last record since 'LogAppendTime' monotonically increases within a partition.