Package org.apache.beam.io.debezium

Class KafkaSourceConsumerFn<T>

java.lang.Object

org.apache.beam.sdk.transforms.DoFn<Map<String,String>,T>

org.apache.beam.io.debezium.KafkaSourceConsumerFn<T>

All Implemented Interfaces:

Serializable, HasDisplayData

public class KafkaSourceConsumerFn<T>
extends DoFn<Map<String,String>,T>

Quick Overview

This is a Splittable DoFn used to process records fetched from supported Debezium Connectors.

Currently it has a time limiter (see KafkaSourceConsumerFn.OffsetTracker) which, if set, it will stop automatically after the specified elapsed minutes. Otherwise, it will keep running until the user explicitly interrupts it.

It might be initialized either as:

KafkaSourceConsumerFn(connectorClass, SourceRecordMapper, maxRecords, millisecondsToRun)
 

Or with a time limiter:

KafkaSourceConsumerFn(connectorClass, SourceRecordMapper, minutesToRun)

See Also:

• Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static class	KafkaSourceConsumerFn.DebeziumSDFDatabaseHistory

Nested classes/interfaces inherited from class org.apache.beam.sdk.transforms.DoFn

DoFn.AlwaysFetched, DoFn.BoundedPerElement, DoFn.BundleFinalizer, DoFn.Element, DoFn.FieldAccess, DoFn.FinishBundle, DoFn.FinishBundleContext, DoFn.GetInitialRestriction, DoFn.GetInitialWatermarkEstimatorState, DoFn.GetRestrictionCoder, DoFn.GetSize, DoFn.GetWatermarkEstimatorStateCoder, DoFn.Key, DoFn.MultiOutputReceiver, DoFn.NewTracker, DoFn.NewWatermarkEstimator, DoFn.OnTimer, DoFn.OnTimerContext, DoFn.OnTimerFamily, DoFn.OnWindowExpiration, DoFn.OnWindowExpirationContext, DoFn.OutputReceiver<T>, DoFn.ProcessContext, DoFn.ProcessContinuation, DoFn.ProcessElement, DoFn.RequiresStableInput, DoFn.RequiresTimeSortedInput, DoFn.Restriction, DoFn.Setup, DoFn.SideInput, DoFn.SplitRestriction, DoFn.StartBundle, DoFn.StartBundleContext, DoFn.StateId, DoFn.Teardown, DoFn.TimerFamily, DoFn.TimerId, DoFn.Timestamp, DoFn.TruncateRestriction, DoFn.UnboundedPerElement, DoFn.WatermarkEstimatorState, DoFn.WindowedContext

Field Summary

Fields

Modifier and Type	Field	Description
static final String	BEAM_INSTANCE_PROPERTY

Method Summary

Modifier and Type	Method	Description
String	getHashCode()
org.apache.beam.io.debezium.KafkaSourceConsumerFn.OffsetHolder	getInitialRestriction(Map<String,String> unused)
Instant	getInitialWatermarkEstimatorState(Instant currentElementTimestamp)
protected org.apache.kafka.connect.source.SourceRecord	getOneRecord(Map<String,String> configuration)
Coder<org.apache.beam.io.debezium.KafkaSourceConsumerFn.OffsetHolder>	getRestrictionCoder()
RestrictionTracker<org.apache.beam.io.debezium.KafkaSourceConsumerFn.OffsetHolder,Map<String,Object>>	newTracker(org.apache.beam.io.debezium.KafkaSourceConsumerFn.OffsetHolder restriction)
WatermarkEstimator<Instant>	newWatermarkEstimator(Instant watermarkEstimatorState)
DoFn.ProcessContinuation	process(Map<String,String> element, RestrictionTracker<org.apache.beam.io.debezium.KafkaSourceConsumerFn.OffsetHolder,Map<String,Object>> tracker, DoFn.OutputReceiver<T> receiver)	Process the retrieved element and format it for output.

Methods inherited from class org.apache.beam.sdk.transforms.DoFn

getAllowedTimestampSkew, getInputTypeDescriptor, getOutputTypeDescriptor, populateDisplayData, prepareForProcessing

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

BEAM_INSTANCE_PROPERTY

public static final String BEAM_INSTANCE_PROPERTY

See Also:

• Constant Field Values

Method Details

getInitialRestriction

@GetInitialRestriction
public org.apache.beam.io.debezium.KafkaSourceConsumerFn.OffsetHolder getInitialRestriction(@Element
 Map<String,String> unused)
                                                                                     throws IOException

Throws:

IOException

newTracker

@NewTracker
public RestrictionTracker<org.apache.beam.io.debezium.KafkaSourceConsumerFn.OffsetHolder,Map<String,Object>> newTracker(@Restriction
 org.apache.beam.io.debezium.KafkaSourceConsumerFn.OffsetHolder restriction)

getRestrictionCoder

@GetRestrictionCoder
public Coder<org.apache.beam.io.debezium.KafkaSourceConsumerFn.OffsetHolder> getRestrictionCoder()

getOneRecord

protected org.apache.kafka.connect.source.SourceRecord getOneRecord(Map<String,String> configuration)

getInitialWatermarkEstimatorState

@GetInitialWatermarkEstimatorState
public Instant getInitialWatermarkEstimatorState(@Timestamp
 Instant currentElementTimestamp)

newWatermarkEstimator

@NewWatermarkEstimator
public WatermarkEstimator<Instant> newWatermarkEstimator(@WatermarkEstimatorState
 Instant watermarkEstimatorState)

process

@ProcessElement
public DoFn.ProcessContinuation process(@Element
 Map<String,String> element,
 RestrictionTracker<org.apache.beam.io.debezium.KafkaSourceConsumerFn.OffsetHolder,Map<String,Object>> tracker,
 DoFn.OutputReceiver<T> receiver)
                                 throws Exception

Process the retrieved element and format it for output. Update all pending

Parameters:

element - A descriptor for the configuration of the SourceConnector and SourceTask instances.

tracker - Restriction Tracker

receiver - Output Receiver

Returns:

DoFn.ProcessContinuation in most cases, to continue processing after 1 second. Otherwise, if we've reached a limit of elements, to stop processing.

Throws:

Exception

getHashCode

public String getHashCode()