public class DoFnFunction<InputT,OutputT> extends java.lang.Object implements org.apache.spark.api.java.function.MapPartitionsFunction<org.apache.beam.sdk.util.WindowedValue<InputT>,scala.Tuple2<TupleTag<?>,org.apache.beam.sdk.util.WindowedValue<?>>>
DoFn
inside a Spark MapPartitionsFunction
.
We get a mapping from TupleTag
to output index and must tag
all outputs with the output number. Afterwards a filter will filter out those elements that are
not to be in a specific output.
Constructor and Description |
---|
DoFnFunction(MetricsContainerStepMapAccumulator metricsAccum,
java.lang.String stepName,
DoFn<InputT,OutputT> doFn,
WindowingStrategy<?,?> windowingStrategy,
java.util.Map<PCollectionView<?>,WindowingStrategy<?,?>> sideInputs,
org.apache.beam.runners.core.construction.SerializablePipelineOptions serializableOptions,
java.util.List<TupleTag<?>> additionalOutputTags,
TupleTag<OutputT> mainOutputTag,
Coder<InputT> inputCoder,
java.util.Map<TupleTag<?>,Coder<?>> outputCoderMap,
SideInputBroadcast broadcastStateData,
DoFnSchemaInformation doFnSchemaInformation,
java.util.Map<java.lang.String,PCollectionView<?>> sideInputMapping) |
Modifier and Type | Method and Description |
---|---|
java.util.Iterator<scala.Tuple2<TupleTag<?>,org.apache.beam.sdk.util.WindowedValue<?>>> |
call(java.util.Iterator<org.apache.beam.sdk.util.WindowedValue<InputT>> iter) |
public DoFnFunction(MetricsContainerStepMapAccumulator metricsAccum, java.lang.String stepName, DoFn<InputT,OutputT> doFn, WindowingStrategy<?,?> windowingStrategy, java.util.Map<PCollectionView<?>,WindowingStrategy<?,?>> sideInputs, org.apache.beam.runners.core.construction.SerializablePipelineOptions serializableOptions, java.util.List<TupleTag<?>> additionalOutputTags, TupleTag<OutputT> mainOutputTag, Coder<InputT> inputCoder, java.util.Map<TupleTag<?>,Coder<?>> outputCoderMap, SideInputBroadcast broadcastStateData, DoFnSchemaInformation doFnSchemaInformation, java.util.Map<java.lang.String,PCollectionView<?>> sideInputMapping)