@Experimental(value=WITH_EXCEPTIONS) public class WithFailures extends java.lang.Object
Consuming transforms such as MapElements.MapWithFailures follow the general pattern of
taking in a user-defined exception handler of type ProcessFunction<ExceptionElement<InputT>, FailureOutputT> where the input WithFailures.ExceptionElement contains an exception along with the input element that was being processed
when the exception was raised. This handler is responsible for producing some output element that
captures relevant details of the failure and can be encoded as part of a failure output PCollection. Transforms can then package together their output and failure collections in a
WithFailures.Result that avoids users needing to interact with TupleTags and
indexing into a PCollectionTuple.
Exception handlers can narrow their scope by rethrowing the passed WithFailures.ExceptionElement.exception() and catching only specific subclasses of Exception.
Unhandled exceptions will generally bubble up to a top-level Pipeline.PipelineExecutionException that halts progress.
Users can take advantage of WithFailures.Result.failuresTo(List) for fluent chaining of transforms
that handle exceptions:
PCollection<Integer> input = ...
List<PCollection<Map<String, String>> failureCollections = new ArrayList<>();
input.apply(MapElements.via(...).exceptionsVia(...))
.failuresTo(failureCollections)
.apply(MapElements.via(...).exceptionsVia(...))
.failuresTo(failureCollections);
PCollection<Map<String, String>> failures = PCollectionList.of(failureCollections)
.apply("FlattenFailureCollections", Flatten.pCollections());
| Modifier and Type | Class and Description |
|---|---|
static class |
WithFailures.ExceptionAsMapHandler<T>
A simple handler that extracts information from an exception to a
Map<String, String>
and returns a KV where the key is the input element that failed processing, and the
value is the map of exception attributes. |
static class |
WithFailures.ExceptionElement<T>
The value type passed as input to exception handlers.
|
static class |
WithFailures.Result<OutputT extends POutput,FailureElementT>
An intermediate output type for PTransforms that allows an output collection to live alongside
a collection of elements that failed the transform.
|
static class |
WithFailures.ThrowableHandler<T>
A handler that holds onto the
Throwable that led to the exception, returning it along
with the original value as a KV. |
| Constructor and Description |
|---|
WithFailures() |