@Internal public class UnboundedSolaceSource<T> extends UnboundedSource<T,SolaceCheckpointMark>
UnboundedSource.CheckpointMark, UnboundedSource.UnboundedReader<OutputT>Source.Reader<T>| Constructor and Description |
|---|
UnboundedSolaceSource(Queue queue,
SempClientFactory sempClientFactory,
SessionServiceFactory sessionServiceFactory,
@Nullable java.lang.Integer maxNumConnections,
boolean enableDeduplication,
Coder<T> coder,
SerializableFunction<T,Instant> timestampFn,
Duration watermarkIdleDurationThreshold,
SerializableFunction<BytesXMLMessage,T> parseFn) |
| Modifier and Type | Method and Description |
|---|---|
UnboundedSource.UnboundedReader<T> |
createReader(PipelineOptions options,
@Nullable SolaceCheckpointMark checkpointMark)
Create a new
UnboundedSource.UnboundedReader to read from this source, resuming from the given
checkpoint if present. |
Coder<SolaceCheckpointMark> |
getCheckpointMarkCoder()
Returns a
Coder for encoding and decoding the checkpoints for this source. |
Coder<T> |
getOutputCoder()
Returns the
Coder to use for the data read from this source. |
SerializableFunction<BytesXMLMessage,T> |
getParseFn() |
Queue |
getQueue() |
SempClientFactory |
getSempClientFactory() |
SessionServiceFactory |
getSessionServiceFactory() |
SerializableFunction<T,Instant> |
getTimestampFn() |
Duration |
getWatermarkIdleDurationThreshold() |
boolean |
requiresDeduping()
Returns whether this source requires explicit deduping.
|
java.util.List<UnboundedSolaceSource<T>> |
split(int desiredNumSplits,
PipelineOptions options)
Returns a list of
UnboundedSource objects representing the instances of this source
that should be used when executing the workflow. |
getDefaultOutputCoder, populateDisplayData, validatepublic UnboundedSolaceSource(Queue queue, SempClientFactory sempClientFactory, SessionServiceFactory sessionServiceFactory, @Nullable java.lang.Integer maxNumConnections, boolean enableDeduplication, Coder<T> coder, SerializableFunction<T,Instant> timestampFn, Duration watermarkIdleDurationThreshold, SerializableFunction<BytesXMLMessage,T> parseFn)
public Queue getQueue()
public SessionServiceFactory getSessionServiceFactory()
public SempClientFactory getSempClientFactory()
public SerializableFunction<T,Instant> getTimestampFn()
public Duration getWatermarkIdleDurationThreshold()
public SerializableFunction<BytesXMLMessage,T> getParseFn()
public UnboundedSource.UnboundedReader<T> createReader(PipelineOptions options, @Nullable SolaceCheckpointMark checkpointMark)
UnboundedSourceUnboundedSource.UnboundedReader to read from this source, resuming from the given
checkpoint if present.createReader in class UnboundedSource<T,SolaceCheckpointMark>public java.util.List<UnboundedSolaceSource<T>> split(int desiredNumSplits, PipelineOptions options) throws java.io.IOException
UnboundedSourceUnboundedSource objects representing the instances of this source
that should be used when executing the workflow. Each split should return a separate partition
of the input data.
For example, for a source reading from a growing directory of files, each split could correspond to a prefix of file names.
Some sources are not splittable, such as reading from a single TCP stream. In that case, only a single split should be returned.
Some data sources automatically partition their data among readers. For these types of
inputs, n identical replicas of the top-level source can be returned.
The size of the returned list should be as close to desiredNumSplits as possible,
but does not have to match exactly. A low number of splits will limit the amount of parallelism
in the source.
split in class UnboundedSource<T,SolaceCheckpointMark>java.io.IOExceptionpublic Coder<SolaceCheckpointMark> getCheckpointMarkCoder()
UnboundedSourceCoder for encoding and decoding the checkpoints for this source.getCheckpointMarkCoder in class UnboundedSource<T,SolaceCheckpointMark>public Coder<T> getOutputCoder()
SourceCoder to use for the data read from this source.getOutputCoder in class Source<T>public boolean requiresDeduping()
UnboundedSourceThis is needed if the underlying data source can return the same record multiple times, such a queuing system with a pull-ack model. Sources where the records read are uniquely identified by the persisted state in the CheckpointMark do not need this.
Generally, if UnboundedSource.CheckpointMark.finalizeCheckpoint() is overridden, this method should
return true. Checkpoint finalization is best-effort, and readers can be resumed from a
checkpoint that has not been finalized.
requiresDeduping in class UnboundedSource<T,SolaceCheckpointMark>