Package org.apache.beam.io.requestresponse

Class RequestResponseIO<RequestT,ResponseT>

java.lang.Object

org.apache.beam.sdk.transforms.PTransform<PCollection<RequestT>,Result<ResponseT>>

org.apache.beam.io.requestresponse.RequestResponseIO<RequestT,ResponseT>

All Implemented Interfaces:

Serializable, HasDisplayData

public class RequestResponseIO<RequestT,ResponseT>
extends PTransform<PCollection<RequestT>,Result<ResponseT>>

PTransform for reading from and writing to Web APIs.

RequestResponseIO is recommended for interacting with external systems that offer RPCs that execute relatively quickly and do not offer advance features to make RPC execution efficient.

For systems that offer features for more efficient reading, for example, tracking progress of RPCs, support for splitting RPCs (deduct two or more RPCs which when combined return the same result), consider using the Apache Beam's `Splittable DoFn` interface instead.

Basic Usage

RequestResponseIO minimally requires implementing the Caller interface:

class MyCaller implements Caller<SomeRequest, SomeResponse> {
    public SomeResponse call(SomeRequest request) throws UserCodeExecutionException {
      // calls the API submitting SomeRequest payload and returning SomeResponse
    }
 }

Then provide RequestResponseIO's of(org.apache.beam.io.requestresponse.Caller<RequestT, ResponseT>, org.apache.beam.sdk.coders.Coder<ResponseT>) method your Caller implementation.

 Coder<SomeResponse> responseCoder = ...
 PCollection<SomeRequest> requests = ...
 Result result = requests.apply(RequestResponseIO.of(new MyCaller(), responseCoder));
 result.getResponses().apply( ... );
 result.getFailures().apply( ... );
 

See Also:

• Serialized Form

Field Summary

Fields

Modifier and Type	Field	Description
static final Duration	DEFAULT_TIMEOUT	The default Duration to wait until completion of user code.
static final Set<Class<? extends UserCodeExecutionException>>	REPEATABLE_ERROR_TYPES	Set of UserCodeExecutionExceptions that warrant repeating.

Fields inherited from class org.apache.beam.sdk.transforms.PTransform

annotations, displayData, name, resourceHints

Method Summary

Modifier and Type	Method	Description
Result<ResponseT>	expand(PCollection<RequestT> input)	Override this method to specify how this PTransform should be expanded on the given InputT.
static <RequestT, ResponseT> RequestResponseIO<RequestT,ResponseT>	of(Caller<RequestT,ResponseT> caller, Coder<ResponseT> responseTCoder)	Instantiates a RequestResponseIO with a Caller and a RequestResponseIO Coder with a default package private implementation of CallShouldBackoff based on https://sre.google/sre-book/handling-overload.
static <RequestT, ResponseT, CallerSetupTeardownT extends Caller<RequestT, ResponseT> & SetupTeardown> RequestResponseIO<RequestT,ResponseT>	ofCallerAndSetupTeardown(CallerSetupTeardownT implementsCallerAndSetupTeardown, Coder<ResponseT> responseTCoder)	Instantiates a RequestResponseIO with a RequestResponseIO Coder, a default package private implementation of CallShouldBackoff based on https://sre.google/sre-book/handling-overload, and an implementation of both the Caller and SetupTeardown interfaces.
RequestResponseIO<RequestT,ResponseT>	withCache(Cache.Pair<RequestT,ResponseT> pair)	Configures RequestResponseIO for reading and writing RequestResponseIO and RequestResponseIO pairs using a cache.
RequestResponseIO<RequestT,ResponseT>	withCallShouldBackoff(CallShouldBackoff<ResponseT> value)	Overrides the package private implementation of CallShouldBackoff, based on https://sre.google/sre-book/handling-overload, that determines whether the underlyling DoFn should hold RequestResponseIOs.
RequestResponseIO<RequestT,ResponseT>	withMonitoringConfiguration(Monitoring value)
RequestResponseIO<RequestT,ResponseT>	withoutRepeater()	Turns off repeat invocations (default is on) of SetupTeardown and Caller, using the Repeater, in the setting of REPEATABLE_ERROR_TYPES.
RequestResponseIO<RequestT,ResponseT>	withTimeout(Duration value)	Overrides the DEFAULT_TIMEOUT expected timeout of all user custom code.

Methods inherited from class org.apache.beam.sdk.transforms.PTransform

addAnnotation, compose, compose, getAdditionalInputs, getAnnotations, getDefaultOutputCoder, getDefaultOutputCoder, getDefaultOutputCoder, getKindString, getName, getResourceHints, populateDisplayData, setDisplayData, setResourceHints, toString, validate, validate

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Details

DEFAULT_TIMEOUT

public static final Duration DEFAULT_TIMEOUT

The default Duration to wait until completion of user code. A UserCodeTimeoutException is thrown when Caller.call(RequestT), SetupTeardown.setup(), or SetupTeardown.teardown() exceed this timeout.

REPEATABLE_ERROR_TYPES

public static final Set<Class<? extends UserCodeExecutionException>> REPEATABLE_ERROR_TYPES

Set of UserCodeExecutionExceptions that warrant repeating. Not all errors should be repeat execution such as bad or unauthenticated requests. However, certain errors such as timeouts, remote system or quota exceed errors may not be related to the code but due to the remote system and thus warrant repeating.

Method Details

of

public static <RequestT,
ResponseT>
RequestResponseIO<RequestT,ResponseT> of(Caller<RequestT,ResponseT> caller,
 Coder<ResponseT> responseTCoder)

Instantiates a RequestResponseIO with a Caller and a RequestResponseIO Coder with a default package private implementation of CallShouldBackoff based on https://sre.google/sre-book/handling-overload. Checks for the Caller's SerializableUtils.ensureSerializable(T) serializable errors.

ofCallerAndSetupTeardown

public static <RequestT,
ResponseT,
CallerSetupTeardownT extends Caller<RequestT,
ResponseT> & SetupTeardown>
RequestResponseIO<RequestT,ResponseT> ofCallerAndSetupTeardown(CallerSetupTeardownT implementsCallerAndSetupTeardown,
 Coder<ResponseT> responseTCoder)

Instantiates a RequestResponseIO with a RequestResponseIO Coder, a default package private implementation of CallShouldBackoff based on https://sre.google/sre-book/handling-overload, and an implementation of both the Caller and SetupTeardown interfaces. Checks SerializableUtils.ensureSerializable(T) serializable errors.

withTimeout

public RequestResponseIO<RequestT,ResponseT> withTimeout(Duration value)

Overrides the DEFAULT_TIMEOUT expected timeout of all user custom code. If user custom code exceeds this timeout, then a UserCodeTimeoutException is thrown. User custom code may throw this exception prior to the configured timeout value on their own.

withoutRepeater

public RequestResponseIO<RequestT,ResponseT> withoutRepeater()

Turns off repeat invocations (default is on) of SetupTeardown and Caller, using the Repeater, in the setting of REPEATABLE_ERROR_TYPES.

withCallShouldBackoff

public RequestResponseIO<RequestT,ResponseT> withCallShouldBackoff(CallShouldBackoff<ResponseT> value)

Overrides the package private implementation of CallShouldBackoff, based on https://sre.google/sre-book/handling-overload, that determines whether the underlyling DoFn should hold RequestResponseIOs.

withCache

public RequestResponseIO<RequestT,ResponseT> withCache(Cache.Pair<RequestT,ResponseT> pair)

Configures RequestResponseIO for reading and writing RequestResponseIO and RequestResponseIO pairs using a cache. RequestResponseIO, by default, does not interact with a cache.

When reading, the transform Flattens the RequestResponseIO PCollection
 of successful pairs with that resulting from API calls of RequestResponseIOs of unsuccessful pairs.
 

When writing, the transform Flattens the RequestResponseIO PCollection of
    successful pairs with that resulting from API calls of RequestResponseIOs.
 

withMonitoringConfiguration

public RequestResponseIO<RequestT,ResponseT> withMonitoringConfiguration(Monitoring value)

expand

public Result<ResponseT> expand(PCollection<RequestT> input)

Description copied from class: PTransform

Override this method to specify how this PTransform should be expanded on the given InputT.

NOTE: This method should not be called directly. Instead apply the PTransform should be applied to the InputT using the apply method.

Composite transforms, which are defined in terms of other transforms, should return the output of one of the composed transforms. Non-composite transforms, which do not apply any transforms internally, should return a new unbound output and register evaluators (via backend-specific registration methods).

Specified by:

expand in class PTransform<PCollection<RequestT>,Result<ResponseT>>