Package org.apache.beam.runners.spark.translation

Class MultiDoFnFunction<InputT,OutputT>

java.lang.Object

org.apache.beam.runners.spark.translation.MultiDoFnFunction<InputT,OutputT>

Type Parameters:

InputT - Input type for DoFunction.

OutputT - Output type for DoFunction.

All Implemented Interfaces:

Serializable, org.apache.spark.api.java.function.PairFlatMapFunction<Iterator<WindowedValue<InputT>>,TupleTag<?>,WindowedValue<?>>

public class MultiDoFnFunction<InputT,OutputT>
extends Object
implements org.apache.spark.api.java.function.PairFlatMapFunction<Iterator<WindowedValue<InputT>>,TupleTag<?>,WindowedValue<?>>

DoFunctions ignore outputs that are not the main output. MultiDoFunctions deal with additional outputs by enriching the underlying data with multiple TupleTags.

See Also:

• Serialized Form

Constructor Summary

Constructors

Constructor

Description

MultiDoFnFunction(MetricsContainerStepMapAccumulator metricsAccum, String stepName, DoFn<InputT,OutputT> doFn, org.apache.beam.runners.core.construction.SerializablePipelineOptions options, TupleTag<OutputT> mainOutputTag, List<TupleTag<?>> additionalOutputTags, Coder<InputT> inputCoder, Map<TupleTag<?>,Coder<?>> outputCoders, Map<TupleTag<?>,KV<WindowingStrategy<?,?>,SideInputBroadcast<?>>> sideInputs, WindowingStrategy<?,?> windowingStrategy, boolean stateful, DoFnSchemaInformation doFnSchemaInformation, Map<String,PCollectionView<?>> sideInputMapping, boolean useBoundedConcurrentOutput, boolean useStreamingSideInput)

Method Summary

Modifier and Type	Method	Description
Iterator<scala.Tuple2<TupleTag<?>,WindowedValue<?>>>	call(Iterator<WindowedValue<InputT>> iter)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

MultiDoFnFunction

public MultiDoFnFunction(MetricsContainerStepMapAccumulator metricsAccum,
 String stepName,
 DoFn<InputT,OutputT> doFn,
 org.apache.beam.runners.core.construction.SerializablePipelineOptions options,
 TupleTag<OutputT> mainOutputTag,
 List<TupleTag<?>> additionalOutputTags,
 Coder<InputT> inputCoder,
 Map<TupleTag<?>,Coder<?>> outputCoders,
 Map<TupleTag<?>,KV<WindowingStrategy<?,?>,SideInputBroadcast<?>>> sideInputs,
 WindowingStrategy<?,?> windowingStrategy,
 boolean stateful,
 DoFnSchemaInformation doFnSchemaInformation,
 Map<String,PCollectionView<?>> sideInputMapping,
 boolean useBoundedConcurrentOutput,
 boolean useStreamingSideInput)

Parameters:

metricsAccum - The Spark AccumulatorV2 that backs the Beam metrics.

doFn - The DoFn to be wrapped.

options - The SerializablePipelineOptions.

mainOutputTag - The main output TupleTag.

additionalOutputTags - Additional output tags.

inputCoder - The coder for the input.

outputCoders - A map of all output coders.

sideInputs - Side inputs used in this DoFn.

windowingStrategy - Input WindowingStrategy.

stateful - Stateful DoFn.

useBoundedConcurrentOutput - If it should use bounded output for processing.

Method Details

call

public Iterator<scala.Tuple2<TupleTag<?>,WindowedValue<?>>> call(Iterator<WindowedValue<InputT>> iter)
                                                          throws Exception

Specified by:

call in interface org.apache.spark.api.java.function.PairFlatMapFunction<Iterator<WindowedValue<InputT>>,TupleTag<?>,WindowedValue<?>>

Throws:

Exception