Package org.apache.beam.sdk.transforms

Class WithFailures

java.lang.Object

org.apache.beam.sdk.transforms.WithFailures

public class WithFailures
extends Object

A collection of utilities for writing transforms that can handle exceptions raised during processing of elements.

Consuming transforms such as MapElements.MapWithFailures follow the general pattern of taking in a user-defined exception handler of type ProcessFunction<ExceptionElement<InputT>, FailureOutputT> where the input WithFailures.ExceptionElement contains an exception along with the input element that was being processed when the exception was raised. This handler is responsible for producing some output element that captures relevant details of the failure and can be encoded as part of a failure output PCollection. Transforms can then package together their output and failure collections in a WithFailures.Result that avoids users needing to interact with TupleTags and indexing into a PCollectionTuple.

Exception handlers can narrow their scope by rethrowing the passed WithFailures.ExceptionElement.exception() and catching only specific subclasses of Exception. Unhandled exceptions will generally bubble up to a top-level Pipeline.PipelineExecutionException that halts progress.

Users can take advantage of WithFailures.Result.failuresTo(List) for fluent chaining of transforms that handle exceptions:

 PCollection<Integer> input = ...
 List<PCollection<Map<String, String>> failureCollections = new ArrayList<>();
 input.apply(MapElements.via(...).exceptionsVia(...))
      .failuresTo(failureCollections)
      .apply(MapElements.via(...).exceptionsVia(...))
      .failuresTo(failureCollections);
 PCollection<Map<String, String>> failures = PCollectionList.of(failureCollections)
      .apply("FlattenFailureCollections", Flatten.pCollections());
 

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static class	WithFailures.ExceptionAsMapHandler<T>	A simple handler that extracts information from an exception to a Map<String, String> and returns a KV where the key is the input element that failed processing, and the value is the map of exception attributes.
static class	WithFailures.ExceptionElement<T>	The value type passed as input to exception handlers.
static class	WithFailures.Result<OutputT extends POutput,FailureElementT>	An intermediate output type for PTransforms that allows an output collection to live alongside a collection of elements that failed the transform.
static class	WithFailures.ThrowableHandler<T>	A handler that holds onto the Throwable that led to the exception, returning it along with the original value as a KV.

Constructor Summary

Constructors

Constructor	Description
WithFailures()

Method Summary

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

WithFailures

public WithFailures()