org.apache.beam.sdk.io.solace.read

Class UnboundedSolaceSource<T>

java.lang.Object

org.apache.beam.sdk.io.Source<OutputT>

org.apache.beam.sdk.io.UnboundedSource<T,SolaceCheckpointMark>

org.apache.beam.sdk.io.solace.read.UnboundedSolaceSource<T>

All Implemented Interfaces:

java.io.Serializable, HasDisplayData

@Internal
public class UnboundedSolaceSource<T>
extends UnboundedSource<T,SolaceCheckpointMark>

See Also:

Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.beam.sdk.io.UnboundedSource

UnboundedSource.CheckpointMark, UnboundedSource.UnboundedReader<OutputT>

Nested classes/interfaces inherited from class org.apache.beam.sdk.io.Source

Source.Reader<T>

Constructor Summary

Constructors

Constructor and Description
UnboundedSolaceSource(com.solacesystems.jcsmp.Queue queue, SempClientFactory sempClientFactory, SessionServiceFactory sessionServiceFactory, @Nullable java.lang.Integer maxNumConnections, boolean enableDeduplication, Coder<T> coder, SerializableFunction<T,Instant> timestampFn, Duration watermarkIdleDurationThreshold, SerializableFunction<com.solacesystems.jcsmp.BytesXMLMessage,T> parseFn)

Method Summary

Modifier and Type	Method and Description
UnboundedSource.UnboundedReader<T>	createReader(PipelineOptions options, @Nullable SolaceCheckpointMark checkpointMark) Create a new UnboundedSource.UnboundedReader to read from this source, resuming from the given checkpoint if present.
Coder<SolaceCheckpointMark>	getCheckpointMarkCoder() Returns a Coder for encoding and decoding the checkpoints for this source.
Coder<T>	getOutputCoder() Returns the Coder to use for the data read from this source.
SerializableFunction<com.solacesystems.jcsmp.BytesXMLMessage,T>	getParseFn()
com.solacesystems.jcsmp.Queue	getQueue()
SempClientFactory	getSempClientFactory()
SessionServiceFactory	getSessionServiceFactory()
SerializableFunction<T,Instant>	getTimestampFn()
Duration	getWatermarkIdleDurationThreshold()
boolean	requiresDeduping() Returns whether this source requires explicit deduping.
java.util.List<UnboundedSolaceSource<T>>	split(int desiredNumSplits, PipelineOptions options) Returns a list of UnboundedSource objects representing the instances of this source that should be used when executing the workflow.

Methods inherited from class org.apache.beam.sdk.io.Source

getDefaultOutputCoder, populateDisplayData, validate

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

UnboundedSolaceSource

public UnboundedSolaceSource(com.solacesystems.jcsmp.Queue queue,
                             SempClientFactory sempClientFactory,
                             SessionServiceFactory sessionServiceFactory,
                             @Nullable java.lang.Integer maxNumConnections,
                             boolean enableDeduplication,
                             Coder<T> coder,
                             SerializableFunction<T,Instant> timestampFn,
                             Duration watermarkIdleDurationThreshold,
                             SerializableFunction<com.solacesystems.jcsmp.BytesXMLMessage,T> parseFn)

Method Detail

getQueue

public com.solacesystems.jcsmp.Queue getQueue()

getSessionServiceFactory

public SessionServiceFactory getSessionServiceFactory()

getSempClientFactory

public SempClientFactory getSempClientFactory()

getTimestampFn

public SerializableFunction<T,Instant> getTimestampFn()

getWatermarkIdleDurationThreshold

public Duration getWatermarkIdleDurationThreshold()

getParseFn

public SerializableFunction<com.solacesystems.jcsmp.BytesXMLMessage,T> getParseFn()

createReader

public UnboundedSource.UnboundedReader<T> createReader(PipelineOptions options,
                                                       @Nullable SolaceCheckpointMark checkpointMark)

Description copied from class: UnboundedSource

Create a new UnboundedSource.UnboundedReader to read from this source, resuming from the given checkpoint if present.

Specified by:

createReader in class UnboundedSource<T,SolaceCheckpointMark>

split

public java.util.List<UnboundedSolaceSource<T>> split(int desiredNumSplits,
                                                      PipelineOptions options)
                                               throws java.io.IOException

Description copied from class: UnboundedSource

Returns a list of UnboundedSource objects representing the instances of this source that should be used when executing the workflow. Each split should return a separate partition of the input data.

For example, for a source reading from a growing directory of files, each split could correspond to a prefix of file names.

Some sources are not splittable, such as reading from a single TCP stream. In that case, only a single split should be returned.

Some data sources automatically partition their data among readers. For these types of inputs, n identical replicas of the top-level source can be returned.

The size of the returned list should be as close to desiredNumSplits as possible, but does not have to match exactly. A low number of splits will limit the amount of parallelism in the source.

Specified by:

split in class UnboundedSource<T,SolaceCheckpointMark>

Throws:

java.io.IOException

getCheckpointMarkCoder

public Coder<SolaceCheckpointMark> getCheckpointMarkCoder()

Description copied from class: UnboundedSource

Returns a Coder for encoding and decoding the checkpoints for this source.

Specified by:

getCheckpointMarkCoder in class UnboundedSource<T,SolaceCheckpointMark>

getOutputCoder

public Coder<T> getOutputCoder()

Description copied from class: Source

Returns the Coder to use for the data read from this source.

Overrides:

getOutputCoder in class Source<T>

requiresDeduping

public boolean requiresDeduping()

Description copied from class: UnboundedSource

Returns whether this source requires explicit deduping.

This is needed if the underlying data source can return the same record multiple times, such a queuing system with a pull-ack model. Sources where the records read are uniquely identified by the persisted state in the CheckpointMark do not need this.

Generally, if UnboundedSource.CheckpointMark.finalizeCheckpoint() is overridden, this method should return true. Checkpoint finalization is best-effort, and readers can be resumed from a checkpoint that has not been finalized.

Overrides:

requiresDeduping in class UnboundedSource<T,SolaceCheckpointMark>