org.apache.beam.sdk.io.solace.write

Class UnboundedStreamingSolaceWriter

java.lang.Object

org.apache.beam.sdk.transforms.DoFn<KV<java.lang.Integer,Solace.Record>,Solace.PublishResult>

org.apache.beam.sdk.io.solace.write.UnboundedSolaceWriter

org.apache.beam.sdk.io.solace.write.UnboundedStreamingSolaceWriter

All Implemented Interfaces:

java.io.Serializable, HasDisplayData

@Internal
public final class UnboundedStreamingSolaceWriter
extends UnboundedSolaceWriter

This DoFn is the responsible for writing to Solace in streaming mode (one message at a time, not holding up any message), and emit the corresponding output (success or fail; only for persistent messages), so the SolaceIO.Write connector can be composed with other subsequent transforms in the pipeline.

The DoFn will create several JCSMP sessions per VM, and the sessions and producers will be reused across different threads (if the number of threads is higher than the number of sessions, which is probably the most common case).

The producer uses the JCSMP streaming mode to publish a single message at a time, processing the acks from this publication, and returning them as output of the DoFn.

There are no acks if the delivery mode is set to DIRECT.

This writer DoFn offers lower latency and lower throughput than UnboundedBatchedSolaceWriter.

See Also:

Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.beam.sdk.transforms.DoFn

DoFn.AlwaysFetched, DoFn.BoundedPerElement, DoFn.BundleFinalizer, DoFn.Element, DoFn.FieldAccess, DoFn.FinishBundle, DoFn.FinishBundleContext, DoFn.GetInitialRestriction, DoFn.GetInitialWatermarkEstimatorState, DoFn.GetRestrictionCoder, DoFn.GetSize, DoFn.GetWatermarkEstimatorStateCoder, DoFn.Key, DoFn.MultiOutputReceiver, DoFn.NewTracker, DoFn.NewWatermarkEstimator, DoFn.OnTimer, DoFn.OnTimerContext, DoFn.OnTimerFamily, DoFn.OnWindowExpiration, DoFn.OnWindowExpirationContext, DoFn.OutputReceiver<T>, DoFn.ProcessContext, DoFn.ProcessContinuation, DoFn.ProcessElement, DoFn.RequiresStableInput, DoFn.RequiresTimeSortedInput, DoFn.Restriction, DoFn.Setup, DoFn.SideInput, DoFn.SplitRestriction, DoFn.StartBundle, DoFn.StartBundleContext, DoFn.StateId, DoFn.Teardown, DoFn.TimerFamily, DoFn.TimerId, DoFn.Timestamp, DoFn.TruncateRestriction, DoFn.UnboundedPerElement, DoFn.WatermarkEstimatorState, DoFn.WindowedContext

Constructor Summary

Constructors

Constructor and Description
UnboundedStreamingSolaceWriter(SerializableFunction<Solace.Record,Destination> destinationFn, SessionServiceFactory sessionServiceFactory, DeliveryMode deliveryMode, SolaceIO.SubmissionMode submissionMode, int producersMapCardinality, boolean publishLatencyMetrics)

Method Summary

Modifier and Type	Method and Description
void	finishBundle(DoFn.FinishBundleContext context)
void	processElement(KV<java.lang.Integer,Solace.Record> element, Instant timestamp, ValueState<java.lang.Integer> currentKeyState)

Methods inherited from class org.apache.beam.sdk.io.solace.write.UnboundedSolaceWriter

addToCurrentBundle, createMessagesArray, createSingleMessage, getCurrentBundle, getCurrentBundleTimestamp, getDeliveryMode, getDestinationFn, getFailedLatencyMetric, getProducersMapCardinality, getPublishLatencyMetric, getSubmissionMode, publishResults, setCurrentBundleTimestamp, shouldPublishLatencyMetrics, solaceSessionServiceWithProducer, startBundle, teardown, updateProducerIndex

Methods inherited from class org.apache.beam.sdk.transforms.DoFn

getAllowedTimestampSkew, getInputTypeDescriptor, getOutputTypeDescriptor, populateDisplayData, prepareForProcessing

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

UnboundedStreamingSolaceWriter

public UnboundedStreamingSolaceWriter(SerializableFunction<Solace.Record,Destination> destinationFn,
                                      SessionServiceFactory sessionServiceFactory,
                                      DeliveryMode deliveryMode,
                                      SolaceIO.SubmissionMode submissionMode,
                                      int producersMapCardinality,
                                      boolean publishLatencyMetrics)

Method Detail

processElement

@DoFn.ProcessElement
public void processElement(@DoFn.Element
                                                KV<java.lang.Integer,Solace.Record> element,
                                                @DoFn.Timestamp
                                                Instant timestamp,
                                                @DoFn.AlwaysFetched @DoFn.StateId(value="current_key")
                                                ValueState<java.lang.Integer> currentKeyState)

finishBundle

@DoFn.FinishBundle
public void finishBundle(DoFn.FinishBundleContext context)