org.apache.beam.sdk.io.kafka

Class TimestampPolicyFactory.LogAppendTimePolicy<K,V>

java.lang.Object

org.apache.beam.sdk.io.kafka.TimestampPolicy<K,V>

org.apache.beam.sdk.io.kafka.TimestampPolicyFactory.LogAppendTimePolicy<K,V>

Enclosing class:

TimestampPolicyFactory<KeyT,ValueT>

public static class TimestampPolicyFactory.LogAppendTimePolicy<K,V>
extends TimestampPolicy<K,V>

Assigns Kafka's log append time (server side ingestion time) to each record. The watermark for each Kafka partition is the timestamp of the last record read. If a partition is idle, the watermark advances roughly to 'current time - 2 seconds'. See KafkaIO.Read.withLogAppendTime() for longer description.

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.beam.sdk.io.kafka.TimestampPolicy

TimestampPolicy.PartitionContext

Field Summary

Fields

Modifier and Type	Field and Description
protected Instant	currentWatermark

Constructor Summary

Constructors

Constructor and Description
LogAppendTimePolicy(java.util.Optional<Instant> previousWatermark)

Method Summary

Modifier and Type	Method and Description
Instant	getTimestampForRecord(TimestampPolicy.PartitionContext context, KafkaRecord<K,V> record) Returns record timestamp (aka event time).
Instant	getWatermark(TimestampPolicy.PartitionContext context) Returns watermark for the partition.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

currentWatermark

protected Instant currentWatermark

Constructor Detail

LogAppendTimePolicy

public LogAppendTimePolicy(java.util.Optional<Instant> previousWatermark)

Method Detail

getTimestampForRecord

public Instant getTimestampForRecord(TimestampPolicy.PartitionContext context,
                                     KafkaRecord<K,V> record)

Description copied from class: TimestampPolicy

Returns record timestamp (aka event time). This is often based on the timestamp of the Kafka record. This is invoked for each record when it is processed in the reader.

Specified by:

getTimestampForRecord in class TimestampPolicy<K,V>

getWatermark

public Instant getWatermark(TimestampPolicy.PartitionContext context)

Description copied from class: TimestampPolicy

Returns watermark for the partition. It is the timestamp before or at the timestamps of all future records consumed from the partition. See UnboundedSource.UnboundedReader#getWatermark() for more guidance on watermarks. E.g. if the record timestamp is 'LogAppendTime', watermark would be the timestamp of the last record since 'LogAppendTime' monotonically increases within a partition.

Specified by:

getWatermark in class TimestampPolicy<K,V>