org.apache.beam.runners.spark.structuredstreaming.translation.batch

Class DoFnFunction<InputT,OutputT>

java.lang.Object

org.apache.beam.runners.spark.structuredstreaming.translation.batch.DoFnFunction<InputT,OutputT>

All Implemented Interfaces:

java.io.Serializable, org.apache.spark.api.java.function.MapPartitionsFunction<org.apache.beam.sdk.util.WindowedValue<InputT>,scala.Tuple2<TupleTag<?>,org.apache.beam.sdk.util.WindowedValue<?>>>

public class DoFnFunction<InputT,OutputT>
extends java.lang.Object
implements org.apache.spark.api.java.function.MapPartitionsFunction<org.apache.beam.sdk.util.WindowedValue<InputT>,scala.Tuple2<TupleTag<?>,org.apache.beam.sdk.util.WindowedValue<?>>>

Encapsulates a DoFn inside a Spark MapPartitionsFunction.

We get a mapping from TupleTag to output index and must tag all outputs with the output number. Afterwards a filter will filter out those elements that are not to be in a specific output.

See Also:

Serialized Form

Constructor Summary

Constructors

Constructor and Description

DoFnFunction(MetricsContainerStepMapAccumulator metricsAccum, java.lang.String stepName, DoFn<InputT,OutputT> doFn, WindowingStrategy<?,?> windowingStrategy, java.util.Map<PCollectionView<?>,WindowingStrategy<?,?>> sideInputs, org.apache.beam.runners.core.construction.SerializablePipelineOptions serializableOptions, java.util.List<TupleTag<?>> additionalOutputTags, TupleTag<OutputT> mainOutputTag, Coder<InputT> inputCoder, java.util.Map<TupleTag<?>,Coder<?>> outputCoderMap, SideInputBroadcast broadcastStateData, DoFnSchemaInformation doFnSchemaInformation, java.util.Map<java.lang.String,PCollectionView<?>> sideInputMapping)

Method Summary

Modifier and Type	Method and Description
java.util.Iterator<scala.Tuple2<TupleTag<?>,org.apache.beam.sdk.util.WindowedValue<?>>>	call(java.util.Iterator<org.apache.beam.sdk.util.WindowedValue<InputT>> iter)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

DoFnFunction

public DoFnFunction(MetricsContainerStepMapAccumulator metricsAccum,
                    java.lang.String stepName,
                    DoFn<InputT,OutputT> doFn,
                    WindowingStrategy<?,?> windowingStrategy,
                    java.util.Map<PCollectionView<?>,WindowingStrategy<?,?>> sideInputs,
                    org.apache.beam.runners.core.construction.SerializablePipelineOptions serializableOptions,
                    java.util.List<TupleTag<?>> additionalOutputTags,
                    TupleTag<OutputT> mainOutputTag,
                    Coder<InputT> inputCoder,
                    java.util.Map<TupleTag<?>,Coder<?>> outputCoderMap,
                    SideInputBroadcast broadcastStateData,
                    DoFnSchemaInformation doFnSchemaInformation,
                    java.util.Map<java.lang.String,PCollectionView<?>> sideInputMapping)

Method Detail

call

public java.util.Iterator<scala.Tuple2<TupleTag<?>,org.apache.beam.sdk.util.WindowedValue<?>>> call(java.util.Iterator<org.apache.beam.sdk.util.WindowedValue<InputT>> iter)
                                                                                             throws java.lang.Exception

Specified by:

call in interface org.apache.spark.api.java.function.MapPartitionsFunction<org.apache.beam.sdk.util.WindowedValue<InputT>,scala.Tuple2<TupleTag<?>,org.apache.beam.sdk.util.WindowedValue<?>>>

Throws:

java.lang.Exception