Package org.apache.beam.sdk.transforms

Class DoFn<InputT extends @Nullable Object,OutputT extends @Nullable Object>

java.lang.Object

org.apache.beam.sdk.transforms.DoFn<InputT,OutputT>

Type Parameters:

InputT - the type of the (main) input elements

OutputT - the type of the (main) output elements

All Implemented Interfaces:

Serializable, HasDisplayData

Direct Known Subclasses:

AddShardKeyDoFn, BatchStatefulParDoOverrides.BatchStatefulDoFn, BeamSetOperatorsTransforms.SetOperatorFilteringDoFn, BeamSqlOutputToConsoleFn, CleanUpReadChangeStreamDoFn, DataflowRunner.StreamingPCollectionViewWriterFn, DataGeneratorRowFn, DetectNewPartitionsDoFn, DetectNewPartitionsDoFn, ErrorHandler.PTransformErrorHandler.WriteErrorMetrics.CountErrors, FhirIO.Deidentify.DeidentifyFn, FhirIO.Export.ExportResourcesFn, FillGaps.FillGapsDoFn, FilterForMutationDoFn, HL7v2IO.HL7v2Read.FetchHL7v2Message.HL7v2MessageGetFn, HL7v2IO.Read.FetchHL7v2Message.HL7v2MessageGetFn, InitializeDoFn, InitializeDoFn, JsonToRow.JsonToRowWithErrFn.ParseWithError, KafkaReadSchemaTransformProvider.ErrorFn, KafkaSourceConsumerFn, PostProcessingMetricsDoFn, PreparePubsubWriteDoFn, PubsubIO.Write.PubsubBoundedWriter, PubsubLiteReadSchemaTransformProvider.ErrorFn, PubsubLiteSink, PubsubLiteWriteSchemaTransformProvider.ErrorCounterFn, PubsubLiteWriteSchemaTransformProvider.SetUuidFromPubSubMessage.SetUuidFn, PubsubWriteSchemaTransformProvider.ErrorFn, RampupThrottlingFn, ReadAllViaFileBasedSourceTransform.AbstractReadFileRangesFn, ReadAllViaFileBasedSourceTransform.SplitIntoRangesFn, ReadChangeStreamPartitionDoFn, ReadChangeStreamPartitionDoFn, ReadFromPulsarDoFn, ReadSpannerSchema, RecordToPublishResultDoFn, Reshuffle.AssignShardFn, SnowflakeIO.Read.CleanTmpFilesFromGcsFn, SnowflakeIO.Read.MapCsvToStringArrayFn, SpannerChangestreamsReadSchemaTransformProvider.DataChangeRecordToRow, SpannerReadSchemaTransformProvider.ErrorFn, StorageApiConvertMessages.ConvertMessagesDoFn, StorageApiFlushAndFinalizeDoFn, TFRecordReadSchemaTransformProvider.ErrorFn, TFRecordWriteSchemaTransformProvider.ErrorFn, UnboundedSolaceWriter, UpdateSchemaDestination, ValueWithRecordId.StripIdsDoFn, View.ToListViewDoFn, Watch.WatchGrowthFn, WriteToPulsarDoFn

public abstract class DoFn<InputT extends @Nullable Object,OutputT extends @Nullable Object>
extends Object
implements Serializable, HasDisplayData

The argument to ParDo providing the code to use to process elements of the input PCollection.

See ParDo for more explanation, examples of use, and discussion of constraints on DoFns, including their serializability, lack of access to global shared mutable state, requirements for failure tolerance, and benefits of optimization.

DoFns can be tested by using TestPipeline. You can verify their functional correctness in a local test using the DirectRunner as well as running integration tests with your production runner of choice. Typically, you can generate the input data using Create.of(java.lang.Iterable<T>) or other transforms. However, if you need to test the behavior of DoFn.StartBundle and DoFn.FinishBundle with particular bundle boundaries, you can use TestStream.

Implementations must define a method annotated with DoFn.ProcessElement that satisfies the requirements described there. See the DoFn.ProcessElement for details.

Example usage:

 PCollection<String> lines = ... ;
 PCollection<String> words =
     lines.apply(ParDo.of(new DoFn<String, String>()) {
         @ProcessElement
          public void processElement(@Element String element, BoundedWindow window) {
            ...
          }}));
 

See Also:

• Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static @interface	DoFn.AlwaysFetched	Annotation for declaring that a state parameter is always fetched.
static @interface	DoFn.BoundedPerElement	Annotation on a splittable DoFn specifying that the DoFn performs a bounded amount of work per input element, so applying it to a bounded PCollection will produce also a bounded PCollection.
static interface	DoFn.BundleFinalizer	A parameter that is accessible during @StartBundle, @ProcessElement and @FinishBundle that allows the caller to register a callback that will be invoked after the bundle has been successfully completed and the runner has commit the output.
static @interface	DoFn.Element	Parameter annotation for the input element for DoFn.ProcessElement, DoFn.GetInitialRestriction, DoFn.GetSize, DoFn.SplitRestriction, DoFn.GetInitialWatermarkEstimatorState, DoFn.NewWatermarkEstimator, and DoFn.NewTracker methods.
static @interface	DoFn.FieldAccess	Annotation for specifying specific fields that are accessed in a Schema PCollection.
static @interface	DoFn.FinishBundle	Annotation for the method to use to finish processing a batch of elements.
class	DoFn.FinishBundleContext	Information accessible while within the DoFn.FinishBundle method.
static @interface	DoFn.GetInitialRestriction	Annotation for the method that maps an element to an initial restriction for a splittable DoFn.
static @interface	DoFn.GetInitialWatermarkEstimatorState	Annotation for the method that maps an element and restriction to initial watermark estimator state for a splittable DoFn.
static @interface	DoFn.GetRestrictionCoder	Annotation for the method that returns the coder to use for the restriction of a splittable DoFn.
static @interface	DoFn.GetSize	Annotation for the method that returns the corresponding size for an element and restriction pair.
static @interface	DoFn.GetWatermarkEstimatorStateCoder	Annotation for the method that returns the coder to use for the watermark estimator state of a splittable DoFn.
static @interface	DoFn.Key	Parameter annotation for dereferencing input element key in KV pair.
static interface	DoFn.MultiOutputReceiver	Receives tagged output for a multi-output function.
static @interface	DoFn.NewTracker	Annotation for the method that creates a new RestrictionTracker for the restriction of a splittable DoFn.
static @interface	DoFn.NewWatermarkEstimator	Annotation for the method that creates a new WatermarkEstimator for the watermark state of a splittable DoFn.
static @interface	DoFn.OnTimer	Annotation for registering a callback for a timer.
class	DoFn.OnTimerContext	Information accessible when running a DoFn.OnTimer method.
static @interface	DoFn.OnTimerFamily	Annotation for registering a callback for a timerFamily.
static @interface	DoFn.OnWindowExpiration	Annotation for the method to use for performing actions on window expiration.
class	DoFn.OnWindowExpirationContext
static interface	DoFn.OutputReceiver<T>	Receives values of the given type.
class	DoFn.ProcessContext	Information accessible when running a DoFn.ProcessElement method.
static class	DoFn.ProcessContinuation	When used as a return value of DoFn.ProcessElement, indicates whether there is more work to be done for the current element.
static @interface	DoFn.ProcessElement	Annotation for the method to use for processing elements.
static @interface	DoFn.RequiresStableInput	Annotation that may be added to a DoFn.ProcessElement, DoFn.OnTimer, or DoFn.OnWindowExpiration method to indicate that the runner must ensure that the observable contents of the input PCollection or mutable state must be stable upon retries.
static @interface	DoFn.RequiresTimeSortedInput	Annotation that may be added to a DoFn.ProcessElement method to indicate that the runner must ensure that the observable contents of the input PCollection is sorted by time, in ascending order.
static @interface	DoFn.Restriction	Parameter annotation for the restriction for DoFn.GetSize, DoFn.SplitRestriction, DoFn.GetInitialWatermarkEstimatorState, DoFn.NewWatermarkEstimator, and DoFn.NewTracker methods.
static @interface	DoFn.Setup	Annotation for the method to use to prepare an instance for processing bundles of elements.
static @interface	DoFn.SideInput	Parameter annotation for the SideInput for a DoFn.ProcessElement method.
static @interface	DoFn.SplitRestriction	Annotation for the method that splits restriction of a splittable DoFn into multiple parts to be processed in parallel.
static @interface	DoFn.StartBundle	Annotation for the method to use to prepare an instance for processing a batch of elements.
class	DoFn.StartBundleContext	Information accessible while within the DoFn.StartBundle method.
static @interface	DoFn.StateId	Annotation for declaring and dereferencing state cells.
static @interface	DoFn.Teardown	Annotation for the method to use to clean up this instance before it is discarded.
static @interface	DoFn.TimerFamily	Parameter annotation for the TimerMap for a DoFn.ProcessElement method.
static @interface	DoFn.TimerId	Annotation for declaring and dereferencing timers.
static @interface	DoFn.Timestamp	Parameter annotation for the input element timestamp for DoFn.ProcessElement, DoFn.GetInitialRestriction, DoFn.GetSize, DoFn.SplitRestriction, DoFn.GetInitialWatermarkEstimatorState, DoFn.NewWatermarkEstimator, and DoFn.NewTracker methods.
static @interface	DoFn.TruncateRestriction	Annotation for the method that truncates the restriction of a splittable DoFn into a bounded one.
static @interface	DoFn.UnboundedPerElement	Annotation on a splittable DoFn specifying that the DoFn performs an unbounded amount of work per input element, so applying it to a bounded PCollection will produce an unbounded PCollection.
static @interface	DoFn.WatermarkEstimatorState	Parameter annotation for the watermark estimator state for the DoFn.NewWatermarkEstimator method.
class	DoFn.WindowedContext	Information accessible to all methods in this DoFn where the context is in some window.

Constructor Summary

Constructors

Constructor	Description
DoFn()

Method Summary

Modifier and Type	Method	Description
Duration	getAllowedTimestampSkew()	Deprecated. This method permits a DoFn to emit elements behind the watermark.
TypeDescriptor<InputT>	getInputTypeDescriptor()	Returns a TypeDescriptor capturing what is known statically about the input type of this DoFn instance's most-derived class.
TypeDescriptor<OutputT>	getOutputTypeDescriptor()	Returns a TypeDescriptor capturing what is known statically about the output type of this DoFn instance's most-derived class.
void	populateDisplayData(DisplayData.Builder builder)	Register display data for the given transform or component.
final void	prepareForProcessing()	Deprecated. use DoFn.Setup or DoFn.StartBundle instead.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

DoFn

public DoFn()

Method Details

getAllowedTimestampSkew

@Deprecated
public Duration getAllowedTimestampSkew()

Deprecated.

This method permits a DoFn to emit elements behind the watermark. These elements are considered late, and if behind the allowed lateness of a downstream PCollection may be silently dropped. See https://github.com/apache/beam/issues/18065 for details on a replacement.

Returns the allowed timestamp skew duration, which is the maximum duration that timestamps can be shifted backward in DoFn.WindowedContext.outputWithTimestamp(OutputT, org.joda.time.Instant).

The default value is Duration.ZERO, in which case timestamps can only be shifted forward to future. For infinite skew, return Duration.millis(Long.MAX_VALUE).

getInputTypeDescriptor

public TypeDescriptor<InputT> getInputTypeDescriptor()

Returns a TypeDescriptor capturing what is known statically about the input type of this DoFn instance's most-derived class.

See getOutputTypeDescriptor() for more discussion.

getOutputTypeDescriptor

public TypeDescriptor<OutputT> getOutputTypeDescriptor()

Returns a TypeDescriptor capturing what is known statically about the output type of this DoFn instance's most-derived class.

In the normal case of a concrete DoFn subclass with no generic type parameters of its own (including anonymous inner classes), this will be a complete non-generic type, which is good for choosing a default output Coder<O> for the output PCollection<O>.

prepareForProcessing

@Deprecated
public final void prepareForProcessing()

Deprecated.

use DoFn.Setup or DoFn.StartBundle instead. This method will be removed in a future release.

Finalize the DoFn construction to prepare for processing. This method should be called by runners before any processing methods.

populateDisplayData

public void populateDisplayData(DisplayData.Builder builder)

Register display data for the given transform or component.

populateDisplayData(DisplayData.Builder) is invoked by Pipeline runners to collect display data via DisplayData.from(HasDisplayData). Implementations may call super.populateDisplayData(builder) in order to register display data in the current namespace, but should otherwise use subcomponent.populateDisplayData(builder) to use the namespace of the subcomponent.

By default, does not register any display data. Implementors may override this method to provide their own display data.

Specified by:

populateDisplayData in interface HasDisplayData

Parameters:

builder - The builder to populate with display data.

See Also:

• HasDisplayData