@Internal public class UnboundedSolaceSource<T> extends UnboundedSource<T,SolaceCheckpointMark>
UnboundedSource.CheckpointMark, UnboundedSource.UnboundedReader<OutputT>
Source.Reader<T>
Constructor and Description |
---|
UnboundedSolaceSource(com.solacesystems.jcsmp.Queue queue,
SempClientFactory sempClientFactory,
SessionServiceFactory sessionServiceFactory,
@Nullable java.lang.Integer maxNumConnections,
boolean enableDeduplication,
Coder<T> coder,
SerializableFunction<T,Instant> timestampFn,
SerializableFunction<com.solacesystems.jcsmp.BytesXMLMessage,T> parseFn) |
Modifier and Type | Method and Description |
---|---|
UnboundedSource.UnboundedReader<T> |
createReader(PipelineOptions options,
@Nullable SolaceCheckpointMark checkpointMark)
Create a new
UnboundedSource.UnboundedReader to read from this source, resuming from the given
checkpoint if present. |
Coder<SolaceCheckpointMark> |
getCheckpointMarkCoder()
Returns a
Coder for encoding and decoding the checkpoints for this source. |
Coder<T> |
getOutputCoder()
Returns the
Coder to use for the data read from this source. |
SerializableFunction<com.solacesystems.jcsmp.BytesXMLMessage,T> |
getParseFn() |
com.solacesystems.jcsmp.Queue |
getQueue() |
SempClientFactory |
getSempClientFactory() |
SessionServiceFactory |
getSessionServiceFactory() |
SerializableFunction<T,Instant> |
getTimestampFn() |
boolean |
requiresDeduping()
Returns whether this source requires explicit deduping.
|
java.util.List<UnboundedSolaceSource<T>> |
split(int desiredNumSplits,
PipelineOptions options)
Returns a list of
UnboundedSource objects representing the instances of this source
that should be used when executing the workflow. |
getDefaultOutputCoder, populateDisplayData, validate
public UnboundedSolaceSource(com.solacesystems.jcsmp.Queue queue, SempClientFactory sempClientFactory, SessionServiceFactory sessionServiceFactory, @Nullable java.lang.Integer maxNumConnections, boolean enableDeduplication, Coder<T> coder, SerializableFunction<T,Instant> timestampFn, SerializableFunction<com.solacesystems.jcsmp.BytesXMLMessage,T> parseFn)
public com.solacesystems.jcsmp.Queue getQueue()
public SessionServiceFactory getSessionServiceFactory()
public SempClientFactory getSempClientFactory()
public SerializableFunction<T,Instant> getTimestampFn()
public SerializableFunction<com.solacesystems.jcsmp.BytesXMLMessage,T> getParseFn()
public UnboundedSource.UnboundedReader<T> createReader(PipelineOptions options, @Nullable SolaceCheckpointMark checkpointMark)
UnboundedSource
UnboundedSource.UnboundedReader
to read from this source, resuming from the given
checkpoint if present.createReader
in class UnboundedSource<T,SolaceCheckpointMark>
public java.util.List<UnboundedSolaceSource<T>> split(int desiredNumSplits, PipelineOptions options) throws java.io.IOException
UnboundedSource
UnboundedSource
objects representing the instances of this source
that should be used when executing the workflow. Each split should return a separate partition
of the input data.
For example, for a source reading from a growing directory of files, each split could correspond to a prefix of file names.
Some sources are not splittable, such as reading from a single TCP stream. In that case, only a single split should be returned.
Some data sources automatically partition their data among readers. For these types of
inputs, n
identical replicas of the top-level source can be returned.
The size of the returned list should be as close to desiredNumSplits
as possible,
but does not have to match exactly. A low number of splits will limit the amount of parallelism
in the source.
split
in class UnboundedSource<T,SolaceCheckpointMark>
java.io.IOException
public Coder<SolaceCheckpointMark> getCheckpointMarkCoder()
UnboundedSource
Coder
for encoding and decoding the checkpoints for this source.getCheckpointMarkCoder
in class UnboundedSource<T,SolaceCheckpointMark>
public Coder<T> getOutputCoder()
Source
Coder
to use for the data read from this source.getOutputCoder
in class Source<T>
public boolean requiresDeduping()
UnboundedSource
This is needed if the underlying data source can return the same record multiple times, such a queuing system with a pull-ack model. Sources where the records read are uniquely identified by the persisted state in the CheckpointMark do not need this.
Generally, if UnboundedSource.CheckpointMark.finalizeCheckpoint()
is overridden, this method should
return true. Checkpoint finalization is best-effort, and readers can be resumed from a
checkpoint that has not been finalized.
requiresDeduping
in class UnboundedSource<T,SolaceCheckpointMark>