org.apache.beam.sdk.io.kafka

Interface TimestampPolicyFactory<KeyT,ValueT>

All Superinterfaces:

java.io.Serializable

Functional Interface:

This is a functional interface and can therefore be used as the assignment target for a lambda expression or method reference.

@FunctionalInterface
public interface TimestampPolicyFactory<KeyT,ValueT>
extends java.io.Serializable

An extendable factory to create a TimestampPolicy for each partition at runtime by KafkaIO reader. Subclasses implement createTimestampPolicy(org.apache.kafka.common.TopicPartition, java.util.Optional<org.joda.time.Instant>), which is invoked by the the reader while starting or resuming from a checkpoint. Two commonly used policies are provided. See withLogAppendTime() and withProcessingTime().

Nested Class Summary

Nested Classes

Modifier and Type	Interface and Description
static class	TimestampPolicyFactory.LogAppendTimePolicy<K,V> Assigns Kafka's log append time (server side ingestion time) to each record.
static class	TimestampPolicyFactory.ProcessingTimePolicy<K,V> A simple policy that uses current time for event time and watermark.
static class	TimestampPolicyFactory.TimestampFnPolicy<K,V> Internal policy to support deprecated withTimestampFn API.

Method Summary

Modifier and Type	Method and Description
TimestampPolicy<KeyT,ValueT>	createTimestampPolicy(org.apache.kafka.common.TopicPartition tp, java.util.Optional<Instant> previousWatermark) Creates a TimestampPolicy for a partition.
static <K,V> TimestampPolicyFactory<K,V>	withCreateTime(Duration maxDelay) CustomTimestampPolicyWithLimitedDelay using KafkaTimestampType.CREATE_TIME from the record for timestamp.
static <K,V> TimestampPolicyFactory<K,V>	withLogAppendTime() A TimestampPolicy that assigns Kafka's log append time (server side ingestion time) to each record.
static <K,V> TimestampPolicyFactory<K,V>	withProcessingTime() A TimestampPolicy that assigns processing time to each record.
static <K,V> TimestampPolicyFactory<K,V>	withTimestampFn(SerializableFunction<KafkaRecord<K,V>,Instant> timestampFn) Used by the Read transform to support old timestamp functions API.

Method Detail

createTimestampPolicy

TimestampPolicy<KeyT,ValueT> createTimestampPolicy(org.apache.kafka.common.TopicPartition tp,
                                                   java.util.Optional<Instant> previousWatermark)

Creates a TimestampPolicy for a partition. This is invoked by the reader at the start or while resuming from previous checkpoint.

Parameters:

tp - The returned policy applies to records from this TopicPartition.

previousWatermark - The latest check-pointed watermark. This is set when the reader is resuming from a checkpoint. This is a good value to return by implementations of TimestampPolicy.getWatermark(PartitionContext) until a better watermark can be established as more records are read.

withProcessingTime

static <K,V> TimestampPolicyFactory<K,V> withProcessingTime()

A TimestampPolicy that assigns processing time to each record. Specifically, this is the timestamp when the record becomes 'current' in the reader. The watermark aways advances to current time.

withLogAppendTime

static <K,V> TimestampPolicyFactory<K,V> withLogAppendTime()

A TimestampPolicy that assigns Kafka's log append time (server side ingestion time) to each record. The watermark for each Kafka partition is the timestamp of the last record read. If a partition is idle, the watermark advances roughly to 'current time - 2 seconds'. See KafkaIO.Read.withLogAppendTime() for longer description.

withCreateTime

static <K,V> TimestampPolicyFactory<K,V> withCreateTime(Duration maxDelay)

CustomTimestampPolicyWithLimitedDelay using KafkaTimestampType.CREATE_TIME from the record for timestamp. See KafkaIO.Read.withCreateTime(Duration) for more complete documentation.

withTimestampFn

static <K,V> TimestampPolicyFactory<K,V> withTimestampFn(SerializableFunction<KafkaRecord<K,V>,Instant> timestampFn)

Used by the Read transform to support old timestamp functions API.