Package org.apache.beam.sdk.io.pulsar

Class ReadFromPulsarDoFn

java.lang.Object

org.apache.beam.sdk.transforms.DoFn<PulsarSourceDescriptor,PulsarMessage>

org.apache.beam.sdk.io.pulsar.ReadFromPulsarDoFn

All Implemented Interfaces:

Serializable, HasDisplayData

@UnboundedPerElement
public class ReadFromPulsarDoFn
extends DoFn<PulsarSourceDescriptor,PulsarMessage>

Transform for reading from Apache Pulsar. Support is currently incomplete, and there may be bugs; see https://github.com/apache/beam/issues/31078 for more info, and comment in that issue if you run into issues with this IO.

See Also:

• Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.beam.sdk.transforms.DoFn

DoFn.AlwaysFetched, DoFn.BoundedPerElement, DoFn.BundleFinalizer, DoFn.Element, DoFn.FieldAccess, DoFn.FinishBundle, DoFn.FinishBundleContext, DoFn.GetInitialRestriction, DoFn.GetInitialWatermarkEstimatorState, DoFn.GetRestrictionCoder, DoFn.GetSize, DoFn.GetWatermarkEstimatorStateCoder, DoFn.Key, DoFn.MultiOutputReceiver, DoFn.NewTracker, DoFn.NewWatermarkEstimator, DoFn.OnTimer, DoFn.OnTimerContext, DoFn.OnTimerFamily, DoFn.OnWindowExpiration, DoFn.OnWindowExpirationContext, DoFn.OutputReceiver<T>, DoFn.ProcessContext, DoFn.ProcessContinuation, DoFn.ProcessElement, DoFn.RequiresStableInput, DoFn.RequiresTimeSortedInput, DoFn.Restriction, DoFn.Setup, DoFn.SideInput, DoFn.SplitRestriction, DoFn.StartBundle, DoFn.StartBundleContext, DoFn.StateId, DoFn.Teardown, DoFn.TimerFamily, DoFn.TimerId, DoFn.Timestamp, DoFn.TruncateRestriction, DoFn.UnboundedPerElement, DoFn.WatermarkEstimatorState, DoFn.WindowedContext

Constructor Summary

Constructors

Constructor	Description
ReadFromPulsarDoFn(PulsarIO.Read transform)

Method Summary

Modifier and Type	Method	Description
OffsetRange	getInitialRestriction(PulsarSourceDescriptor pulsarSource)
Instant	getInitialWatermarkEstimatorState(Instant currentElementTimestamp)
Coder<OffsetRange>	getRestrictionCoder()
double	getSize(PulsarSourceDescriptor pulsarSource, OffsetRange range)
void	initPulsarClients()
org.apache.pulsar.client.api.Reader<byte[]>	newReader(org.apache.pulsar.client.api.PulsarClient client, String topicPartition)
WatermarkEstimator<Instant>	newWatermarkEstimator(Instant watermarkEstimatorState)
DoFn.ProcessContinuation	processElement(PulsarSourceDescriptor pulsarSourceDescriptor, RestrictionTracker<OffsetRange,Long> tracker, WatermarkEstimator watermarkEstimator, DoFn.OutputReceiver<PulsarMessage> output)
OffsetRangeTracker	restrictionTracker(PulsarSourceDescriptor pulsarSource, OffsetRange restriction)
void	teardown()

Methods inherited from class org.apache.beam.sdk.transforms.DoFn

getAllowedTimestampSkew, getInputTypeDescriptor, getOutputTypeDescriptor, populateDisplayData, prepareForProcessing

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

ReadFromPulsarDoFn

public ReadFromPulsarDoFn(PulsarIO.Read transform)

Method Details

initPulsarClients

@Setup
public void initPulsarClients()
                       throws Exception

Throws:

Exception

teardown

@Teardown
public void teardown()
              throws Exception

Throws:

Exception

getInitialRestriction

@GetInitialRestriction
public OffsetRange getInitialRestriction(@Element
 PulsarSourceDescriptor pulsarSource)

getSize

@GetSize
public double getSize(@Element
 PulsarSourceDescriptor pulsarSource,
 @Restriction
 OffsetRange range)

newReader

public org.apache.pulsar.client.api.Reader<byte[]> newReader(org.apache.pulsar.client.api.PulsarClient client,
 String topicPartition)
                                                      throws org.apache.pulsar.client.api.PulsarClientException

Throws:

org.apache.pulsar.client.api.PulsarClientException

getRestrictionCoder

@GetRestrictionCoder
public Coder<OffsetRange> getRestrictionCoder()

processElement

@ProcessElement
public DoFn.ProcessContinuation processElement(@Element
 PulsarSourceDescriptor pulsarSourceDescriptor,
 RestrictionTracker<OffsetRange,Long> tracker,
 WatermarkEstimator watermarkEstimator,
 DoFn.OutputReceiver<PulsarMessage> output)
                                        throws IOException

Throws:

IOException

getInitialWatermarkEstimatorState

@GetInitialWatermarkEstimatorState
public Instant getInitialWatermarkEstimatorState(@Timestamp
 Instant currentElementTimestamp)

newWatermarkEstimator

@NewWatermarkEstimator
public WatermarkEstimator<Instant> newWatermarkEstimator(@WatermarkEstimatorState
 Instant watermarkEstimatorState)

restrictionTracker

@NewTracker
public OffsetRangeTracker restrictionTracker(@Element
 PulsarSourceDescriptor pulsarSource,
 @Restriction
 OffsetRange restriction)