org.apache.beam.sdk.io.kafka

Class TimestampPolicyFactory<KeyT,ValueT>

java.lang.Object

org.apache.beam.sdk.io.kafka.TimestampPolicyFactory<KeyT,ValueT>

All Implemented Interfaces:

java.io.Serializable

public abstract class TimestampPolicyFactory<KeyT,ValueT>
extends java.lang.Object
implements java.io.Serializable

An extendable factory to create a TimestampPolicy for each partition at runtime by KafkaIO reader. Subclasses implement createTimestampPolicy(TopicPartition, java.util.Optional<org.joda.time.Instant>), which is invoked by the the reader while starting or resuming from a checkpoint. Two commonly used policies are provided. See withLogAppendTime() and withProcessingTime().

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	TimestampPolicyFactory.LogAppendTimePolicy<K,V> Assigns Kafka's log append time (server side ingestion time) to each record.
static class	TimestampPolicyFactory.ProcessingTimePolicy<K,V> A simple policy that uses current time for event time and watermark.

Constructor Summary

Constructors

Constructor and Description
TimestampPolicyFactory()

Method Summary

Modifier and Type	Method and Description
abstract TimestampPolicy<KeyT,ValueT>	createTimestampPolicy(TopicPartition tp, java.util.Optional<Instant> previousWatermark) Creates a TimestampPolicy for a partition.
static <K,V> TimestampPolicyFactory<K,V>	withLogAppendTime() A TimestampPolicy that assigns Kafka's log append time (server side ingestion time) to each record.
static <K,V> TimestampPolicyFactory<K,V>	withProcessingTime() A TimestampPolicy that assigns processing time to each record.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

TimestampPolicyFactory

public TimestampPolicyFactory()

Method Detail

createTimestampPolicy

public abstract TimestampPolicy<KeyT,ValueT> createTimestampPolicy(TopicPartition tp,
                                                                   java.util.Optional<Instant> previousWatermark)

Creates a TimestampPolicy for a partition. This is invoked by the reader at the start or while resuming from previous checkpoint.

Parameters:

tp - The returned policy applies to records from this TopicPartition.

previousWatermark - The latest check-pointed watermark. This is set when the reader is resuming from a checkpoint. This is a good value to return by implementations of TimestampPolicy.getWatermark(PartitionContext) until a better watermark can be established as more records are read.

Returns:

withProcessingTime

public static <K,V> TimestampPolicyFactory<K,V> withProcessingTime()

A TimestampPolicy that assigns processing time to each record. Specifically, this is the timestamp when the record becomes 'current' in the reader. The watermark aways advances to current time.

withLogAppendTime

public static <K,V> TimestampPolicyFactory<K,V> withLogAppendTime()

A TimestampPolicy that assigns Kafka's log append time (server side ingestion time) to each record. The watermark for each Kafka partition is the timestamp of the last record read. If a partition is idle, the watermark advances roughly to 'current time - 2 seconds'. See KafkaIO.Read.withLogAppendTime() for longer description.