org.apache.beam.sdk.io.pulsar

Class ReadFromPulsarDoFn

java.lang.Object

org.apache.beam.sdk.transforms.DoFn<PulsarSourceDescriptor,PulsarMessage>

org.apache.beam.sdk.io.pulsar.ReadFromPulsarDoFn

All Implemented Interfaces:

java.io.Serializable, HasDisplayData

@DoFn.UnboundedPerElement
public class ReadFromPulsarDoFn
extends DoFn<PulsarSourceDescriptor,PulsarMessage>

See Also:

Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.beam.sdk.transforms.DoFn

DoFn.AlwaysFetched, DoFn.BoundedPerElement, DoFn.BundleFinalizer, DoFn.Element, DoFn.FieldAccess, DoFn.FinishBundle, DoFn.FinishBundleContext, DoFn.GetInitialRestriction, DoFn.GetInitialWatermarkEstimatorState, DoFn.GetRestrictionCoder, DoFn.GetSize, DoFn.GetWatermarkEstimatorStateCoder, DoFn.Key, DoFn.MultiOutputReceiver, DoFn.NewTracker, DoFn.NewWatermarkEstimator, DoFn.OnTimer, DoFn.OnTimerContext, DoFn.OnTimerFamily, DoFn.OnWindowExpiration, DoFn.OnWindowExpirationContext, DoFn.OutputReceiver<T>, DoFn.ProcessContext, DoFn.ProcessContinuation, DoFn.ProcessElement, DoFn.RequiresStableInput, DoFn.RequiresTimeSortedInput, DoFn.Restriction, DoFn.Setup, DoFn.SideInput, DoFn.SplitRestriction, DoFn.StartBundle, DoFn.StartBundleContext, DoFn.StateId, DoFn.Teardown, DoFn.TimerFamily, DoFn.TimerId, DoFn.Timestamp, DoFn.TruncateRestriction, DoFn.UnboundedPerElement, DoFn.WatermarkEstimatorState, DoFn.WindowedContext

Constructor Summary

Constructors

Constructor and Description
ReadFromPulsarDoFn(PulsarIO.Read transform)

Method Summary

Modifier and Type	Method and Description
OffsetRange	getInitialRestriction(PulsarSourceDescriptor pulsarSource)
Instant	getInitialWatermarkEstimatorState(Instant currentElementTimestamp)
Coder<OffsetRange>	getRestrictionCoder()
double	getSize(PulsarSourceDescriptor pulsarSource, OffsetRange range)
void	initPulsarClients()
org.apache.pulsar.client.api.Reader<byte[]>	newReader(org.apache.pulsar.client.api.PulsarClient client, java.lang.String topicPartition)
WatermarkEstimator<Instant>	newWatermarkEstimator(Instant watermarkEstimatorState)
DoFn.ProcessContinuation	processElement(PulsarSourceDescriptor pulsarSourceDescriptor, RestrictionTracker<OffsetRange,java.lang.Long> tracker, WatermarkEstimator watermarkEstimator, DoFn.OutputReceiver<PulsarMessage> output)
OffsetRangeTracker	restrictionTracker(PulsarSourceDescriptor pulsarSource, OffsetRange restriction)
void	teardown()

Methods inherited from class org.apache.beam.sdk.transforms.DoFn

getAllowedTimestampSkew, getInputTypeDescriptor, getOutputTypeDescriptor, populateDisplayData, prepareForProcessing

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

ReadFromPulsarDoFn

public ReadFromPulsarDoFn(PulsarIO.Read transform)

Method Detail

initPulsarClients

@DoFn.Setup
public void initPulsarClients()
                                   throws java.lang.Exception

Throws:

java.lang.Exception

teardown

@DoFn.Teardown
public void teardown()
                             throws java.lang.Exception

Throws:

java.lang.Exception

getInitialRestriction

@DoFn.GetInitialRestriction
public OffsetRange getInitialRestriction(@DoFn.Element
                                                                     PulsarSourceDescriptor pulsarSource)

getSize

@DoFn.GetSize
public double getSize(@DoFn.Element
                                    PulsarSourceDescriptor pulsarSource,
                                    @DoFn.Restriction
                                    OffsetRange range)

newReader

public org.apache.pulsar.client.api.Reader<byte[]> newReader(org.apache.pulsar.client.api.PulsarClient client,
                                                             java.lang.String topicPartition)
                                                      throws org.apache.pulsar.client.api.PulsarClientException

Throws:

org.apache.pulsar.client.api.PulsarClientException

getRestrictionCoder

@DoFn.GetRestrictionCoder
public Coder<OffsetRange> getRestrictionCoder()

processElement

@DoFn.ProcessElement
public DoFn.ProcessContinuation processElement(@DoFn.Element
                                                                    PulsarSourceDescriptor pulsarSourceDescriptor,
                                                                    RestrictionTracker<OffsetRange,java.lang.Long> tracker,
                                                                    WatermarkEstimator watermarkEstimator,
                                                                    DoFn.OutputReceiver<PulsarMessage> output)
                                                             throws java.io.IOException

Throws:

java.io.IOException

getInitialWatermarkEstimatorState

@DoFn.GetInitialWatermarkEstimatorState
public Instant getInitialWatermarkEstimatorState(@DoFn.Timestamp
                                                                                         Instant currentElementTimestamp)

newWatermarkEstimator

@DoFn.NewWatermarkEstimator
public WatermarkEstimator<Instant> newWatermarkEstimator(@DoFn.WatermarkEstimatorState
                                                                                     Instant watermarkEstimatorState)

restrictionTracker

@DoFn.NewTracker
public OffsetRangeTracker restrictionTracker(@DoFn.Element
                                                              PulsarSourceDescriptor pulsarSource,
                                                              @DoFn.Restriction
                                                              OffsetRange restriction)