Package org.apache.beam.sdk.io.pulsar

Class NaiveReadFromPulsarDoFn<T>

java.lang.Object

org.apache.beam.sdk.transforms.DoFn<PulsarSourceDescriptor,T>

org.apache.beam.sdk.io.pulsar.NaiveReadFromPulsarDoFn<T>

All Implemented Interfaces:

Serializable, HasDisplayData

@UnboundedPerElement
public class NaiveReadFromPulsarDoFn<T>
extends DoFn<PulsarSourceDescriptor,T>

DoFn for reading from Apache Pulsar based on Pulsar Reader from the start message id. It does not support split or acknowledge message get read.

See Also:

• Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.beam.sdk.transforms.DoFn

DoFn.AlwaysFetched, DoFn.BoundedPerElement, DoFn.BundleFinalizer, DoFn.Element, DoFn.FieldAccess, DoFn.FinishBundle, DoFn.FinishBundleContext, DoFn.GetInitialRestriction, DoFn.GetInitialWatermarkEstimatorState, DoFn.GetRestrictionCoder, DoFn.GetSize, DoFn.GetWatermarkEstimatorStateCoder, DoFn.Key, DoFn.MultiOutputReceiver, DoFn.NewTracker, DoFn.NewWatermarkEstimator, DoFn.OnTimer, DoFn.OnTimerContext, DoFn.OnTimerFamily, DoFn.OnWindowExpiration, DoFn.OnWindowExpirationContext, DoFn.OutputReceiver<T>, DoFn.ProcessContext, DoFn.ProcessContinuation, DoFn.ProcessElement, DoFn.RequiresStableInput, DoFn.RequiresTimeSortedInput, DoFn.Restriction, DoFn.Setup, DoFn.SideInput, DoFn.SplitRestriction, DoFn.StartBundle, DoFn.StartBundleContext, DoFn.StateId, DoFn.Teardown, DoFn.TimerFamily, DoFn.TimerId, DoFn.Timestamp, DoFn.TruncateRestriction, DoFn.UnboundedPerElement, DoFn.WatermarkEstimatorState, DoFn.WindowedContext

Constructor Summary

Constructors

Constructor	Description
NaiveReadFromPulsarDoFn(PulsarIO.Read<T> transform)

Method Summary

Modifier and Type	Method	Description
OffsetRange	getInitialRestriction(PulsarSourceDescriptor pulsarSource)
Instant	getInitialWatermarkEstimatorState(Instant currentElementTimestamp)
Coder<OffsetRange>	getRestrictionCoder()
double	getSize(PulsarSourceDescriptor pulsarSource, OffsetRange range)
void	initPulsarClients()	Open connection to Pulsar clients.
org.apache.pulsar.client.api.Reader<byte[]>	newReader(org.apache.pulsar.client.api.PulsarClient client, String topicPartition)
WatermarkEstimator<Instant>	newWatermarkEstimator(Instant watermarkEstimatorState)
DoFn.ProcessContinuation	processElement(PulsarSourceDescriptor pulsarSourceDescriptor, RestrictionTracker<OffsetRange,Long> tracker, WatermarkEstimator<Instant> watermarkEstimator, DoFn.OutputReceiver<T> output)
OffsetRangeTracker	restrictionTracker(PulsarSourceDescriptor pulsarSource, OffsetRange restriction)
void	splitRestriction(OffsetRange restriction, DoFn.OutputReceiver<OffsetRange> receiver, PipelineOptions unused)
void	teardown()	Close connection to Pulsar clients.

Methods inherited from class org.apache.beam.sdk.transforms.DoFn

getAllowedTimestampSkew, getInputTypeDescriptor, getOutputTypeDescriptor, populateDisplayData, prepareForProcessing

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

NaiveReadFromPulsarDoFn

public NaiveReadFromPulsarDoFn(PulsarIO.Read<T> transform)

Method Details

initPulsarClients

@Setup
public void initPulsarClients()
                       throws Exception

Open connection to Pulsar clients.

Throws:

Exception

teardown

@Teardown
public void teardown()
              throws Exception

Close connection to Pulsar clients.

Throws:

Exception

getInitialRestriction

@GetInitialRestriction
public OffsetRange getInitialRestriction(@Element
 PulsarSourceDescriptor pulsarSource)

getSize

@GetSize
public double getSize(@Element
 PulsarSourceDescriptor pulsarSource,
 @Restriction
 OffsetRange range)

newReader

public org.apache.pulsar.client.api.Reader<byte[]> newReader(org.apache.pulsar.client.api.PulsarClient client,
 String topicPartition)
                                                      throws org.apache.pulsar.client.api.PulsarClientException

Throws:

org.apache.pulsar.client.api.PulsarClientException

getRestrictionCoder

@GetRestrictionCoder
public Coder<OffsetRange> getRestrictionCoder()

processElement

@ProcessElement
public DoFn.ProcessContinuation processElement(@Element
 PulsarSourceDescriptor pulsarSourceDescriptor,
 RestrictionTracker<OffsetRange,Long> tracker,
 WatermarkEstimator<Instant> watermarkEstimator,
 DoFn.OutputReceiver<T> output)
                                        throws IOException

Throws:

IOException

splitRestriction

@SplitRestriction
public void splitRestriction(@Restriction
 OffsetRange restriction,
 DoFn.OutputReceiver<OffsetRange> receiver,
 PipelineOptions unused)

getInitialWatermarkEstimatorState

@GetInitialWatermarkEstimatorState
public Instant getInitialWatermarkEstimatorState(@Timestamp
 Instant currentElementTimestamp)

newWatermarkEstimator

@NewWatermarkEstimator
public WatermarkEstimator<Instant> newWatermarkEstimator(@WatermarkEstimatorState
 Instant watermarkEstimatorState)

restrictionTracker

@NewTracker
public OffsetRangeTracker restrictionTracker(@Element
 PulsarSourceDescriptor pulsarSource,
 @Restriction
 OffsetRange restriction)