org.apache.beam.io.debezium

Class KafkaSourceConsumerFn<T>

java.lang.Object

org.apache.beam.sdk.transforms.DoFn<java.util.Map<java.lang.String,java.lang.String>,T>

org.apache.beam.io.debezium.KafkaSourceConsumerFn<T>

All Implemented Interfaces:

java.io.Serializable, HasDisplayData

public class KafkaSourceConsumerFn<T>
extends DoFn<java.util.Map<java.lang.String,java.lang.String>,T>

Quick Overview

This is a Splittable DoFn used to process records fetched from supported Debezium Connectors.

Currently it has a time limiter (see OffsetTracker) which, if set, it will stop automatically after the specified elapsed minutes. Otherwise, it will keep running until the user explicitly interrupts it.

It might be initialized either as:

KafkaSourceConsumerFn(connectorClass, SourceRecordMapper, maxRecords, milisecondsToRun)
 

Or with a time limiter:

KafkaSourceConsumerFn(connectorClass, SourceRecordMapper, minutesToRun)

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	KafkaSourceConsumerFn.DebeziumSDFDatabaseHistory

Nested classes/interfaces inherited from class org.apache.beam.sdk.transforms.DoFn

DoFn.AlwaysFetched, DoFn.BoundedPerElement, DoFn.BundleFinalizer, DoFn.Element, DoFn.FieldAccess, DoFn.FinishBundle, DoFn.FinishBundleContext, DoFn.GetInitialRestriction, DoFn.GetInitialWatermarkEstimatorState, DoFn.GetRestrictionCoder, DoFn.GetSize, DoFn.GetWatermarkEstimatorStateCoder, DoFn.Key, DoFn.MultiOutputReceiver, DoFn.NewTracker, DoFn.NewWatermarkEstimator, DoFn.OnTimer, DoFn.OnTimerContext, DoFn.OnTimerFamily, DoFn.OnWindowExpiration, DoFn.OnWindowExpirationContext, DoFn.OutputReceiver<T>, DoFn.ProcessContext, DoFn.ProcessContinuation, DoFn.ProcessElement, DoFn.RequiresStableInput, DoFn.RequiresTimeSortedInput, DoFn.Restriction, DoFn.Setup, DoFn.SideInput, DoFn.SplitRestriction, DoFn.StartBundle, DoFn.StartBundleContext, DoFn.StateId, DoFn.Teardown, DoFn.TimerFamily, DoFn.TimerId, DoFn.Timestamp, DoFn.TruncateRestriction, DoFn.UnboundedPerElement, DoFn.WatermarkEstimatorState, DoFn.WindowedContext

Field Summary

Fields

Modifier and Type	Field and Description
static java.lang.String	BEAM_INSTANCE_PROPERTY

Method Summary

Modifier and Type	Method and Description
java.lang.String	getHashCode()
org.apache.beam.io.debezium.KafkaSourceConsumerFn.OffsetHolder	getInitialRestriction(java.util.Map<java.lang.String,java.lang.String> unused)
Instant	getInitialWatermarkEstimatorState(Instant currentElementTimestamp)
protected org.apache.kafka.connect.source.SourceRecord	getOneRecord(java.util.Map<java.lang.String,java.lang.String> configuration)
Coder<org.apache.beam.io.debezium.KafkaSourceConsumerFn.OffsetHolder>	getRestrictionCoder()
RestrictionTracker<org.apache.beam.io.debezium.KafkaSourceConsumerFn.OffsetHolder,java.util.Map<java.lang.String,java.lang.Object>>	newTracker(org.apache.beam.io.debezium.KafkaSourceConsumerFn.OffsetHolder restriction)
WatermarkEstimator<Instant>	newWatermarkEstimator(Instant watermarkEstimatorState)
DoFn.ProcessContinuation	process(java.util.Map<java.lang.String,java.lang.String> element, RestrictionTracker<org.apache.beam.io.debezium.KafkaSourceConsumerFn.OffsetHolder,java.util.Map<java.lang.String,java.lang.Object>> tracker, DoFn.OutputReceiver<T> receiver) Process the retrieved element and format it for output.

Methods inherited from class org.apache.beam.sdk.transforms.DoFn

getAllowedTimestampSkew, getInputTypeDescriptor, getOutputTypeDescriptor, populateDisplayData, prepareForProcessing

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

BEAM_INSTANCE_PROPERTY

public static final java.lang.String BEAM_INSTANCE_PROPERTY

See Also:

Constant Field Values

Method Detail

getInitialRestriction

@DoFn.GetInitialRestriction
public org.apache.beam.io.debezium.KafkaSourceConsumerFn.OffsetHolder getInitialRestriction(@DoFn.Element
                                                                                                                        java.util.Map<java.lang.String,java.lang.String> unused)
                                                                                                                 throws java.io.IOException

Throws:

java.io.IOException

newTracker

@DoFn.NewTracker
public RestrictionTracker<org.apache.beam.io.debezium.KafkaSourceConsumerFn.OffsetHolder,java.util.Map<java.lang.String,java.lang.Object>> newTracker(@DoFn.Restriction
                                                                                                                                                                       org.apache.beam.io.debezium.KafkaSourceConsumerFn.OffsetHolder restriction)

getRestrictionCoder

@DoFn.GetRestrictionCoder
public Coder<org.apache.beam.io.debezium.KafkaSourceConsumerFn.OffsetHolder> getRestrictionCoder()

getOneRecord

protected org.apache.kafka.connect.source.SourceRecord getOneRecord(java.util.Map<java.lang.String,java.lang.String> configuration)

getInitialWatermarkEstimatorState

@DoFn.GetInitialWatermarkEstimatorState
public Instant getInitialWatermarkEstimatorState(@DoFn.Timestamp
                                                                                         Instant currentElementTimestamp)

newWatermarkEstimator

@DoFn.NewWatermarkEstimator
public WatermarkEstimator<Instant> newWatermarkEstimator(@DoFn.WatermarkEstimatorState
                                                                                     Instant watermarkEstimatorState)

process

@DoFn.ProcessElement
public DoFn.ProcessContinuation process(@DoFn.Element
                                                             java.util.Map<java.lang.String,java.lang.String> element,
                                                             RestrictionTracker<org.apache.beam.io.debezium.KafkaSourceConsumerFn.OffsetHolder,java.util.Map<java.lang.String,java.lang.Object>> tracker,
                                                             DoFn.OutputReceiver<T> receiver)
                                                      throws java.lang.Exception

Process the retrieved element and format it for output. Update all pending

Parameters:

element - A descriptor for the configuration of the SourceConnector and SourceTask instances.

tracker - Restriction Tracker

receiver - Output Receiver

Returns:

DoFn.ProcessContinuation in most cases, to continue processing after 1 second. Otherwise, if we've reached a limit of elements, to stop processing.

Throws:

java.lang.Exception

getHashCode

public java.lang.String getHashCode()