org.apache.beam.io.requestresponse

Class RequestResponseIO<RequestT,ResponseT>

java.lang.Object

org.apache.beam.sdk.transforms.PTransform<PCollection<RequestT>,Result<ResponseT>>

org.apache.beam.io.requestresponse.RequestResponseIO<RequestT,ResponseT>

All Implemented Interfaces:

java.io.Serializable, HasDisplayData

public class RequestResponseIO<RequestT,ResponseT>
extends PTransform<PCollection<RequestT>,Result<ResponseT>>

PTransform for reading from and writing to Web APIs.

RequestResponseIO is recommended for interacting with external systems that offer RPCs that execute relatively quickly and do not offer advance features to make RPC execution efficient.

For systems that offer features for more efficient reading, for example, tracking progress of RPCs, support for splitting RPCs (deduct two or more RPCs which when combined return the same result), consider using the Apache Beam's `Splittable DoFn` interface instead.

Basic Usage

RequestResponseIO minimally requires implementing the Caller interface:

 class MyCaller implements Caller<SomeRequest, SomeResponse> {
    public SomeResponse call(SomeRequest request) throws UserCodeExecutionException {
      // calls the API submitting SomeRequest payload and returning SomeResponse
    }
 }

Then provide RequestResponseIO's of(org.apache.beam.io.requestresponse.Caller<RequestT, ResponseT>, org.apache.beam.sdk.coders.Coder<ResponseT>) method your Caller implementation.

 Coder<SomeResponse> responseCoder = ...
 PCollection<SomeRequest> requests = ...
 Result result = requests.apply(RequestResponseIO.of(new MyCaller(), responseCoder));
 result.getResponses().apply( ... );
 result.getFailures().apply( ... );
 

See Also:

Serialized Form

Field Summary

Fields

Modifier and Type	Field and Description
static Duration	DEFAULT_TIMEOUT The default Duration to wait until completion of user code.
static java.util.Set<java.lang.Class<? extends UserCodeExecutionException>>	REPEATABLE_ERROR_TYPES Set of UserCodeExecutionExceptions that warrant repeating.

Fields inherited from class org.apache.beam.sdk.transforms.PTransform

annotations, displayData, name, resourceHints

Method Summary

Modifier and Type	Method and Description
Result<ResponseT>	expand(PCollection<RequestT> input) Override this method to specify how this PTransform should be expanded on the given InputT.
static <RequestT,ResponseT> RequestResponseIO<RequestT,ResponseT>	of(Caller<RequestT,ResponseT> caller, Coder<ResponseT> responseTCoder) Instantiates a RequestResponseIO with a Caller and a ResponseT Coder.
static <RequestT,ResponseT,CallerSetupTeardownT extends Caller<RequestT,ResponseT> & SetupTeardown> RequestResponseIO<RequestT,ResponseT>	ofCallerAndSetupTeardown(CallerSetupTeardownT implementsCallerAndSetupTeardown, Coder<ResponseT> responseTCoder) Instantiates a RequestResponseIO with a ResponseT Coder and an implementation of both the Caller and SetupTeardown interfaces.
RequestResponseIO<RequestT,ResponseT>	withCache(Cache.Pair<RequestT,ResponseT> pair) Configures RequestResponseIO for reading and writing RequestT and ResponseT pairs using a cache.
RequestResponseIO<RequestT,ResponseT>	withCallShouldBackoff(CallShouldBackoff<ResponseT> value) Overrides the private no-op implementation of CallShouldBackoff that determines whether the DoFn should hold RequestTs.
RequestResponseIO<RequestT,ResponseT>	withMonitoringConfiguration(Monitoring value)
RequestResponseIO<RequestT,ResponseT>	withoutRepeater() Turns off repeat invocations (default is on) of SetupTeardown and Caller, using the Repeater, in the setting of REPEATABLE_ERROR_TYPES.
RequestResponseIO<RequestT,ResponseT>	withTimeout(Duration value) Overrides the DEFAULT_TIMEOUT expected timeout of all user custom code.

Methods inherited from class org.apache.beam.sdk.transforms.PTransform

addAnnotation, compose, compose, getAdditionalInputs, getAnnotations, getDefaultOutputCoder, getDefaultOutputCoder, getDefaultOutputCoder, getKindString, getName, getResourceHints, populateDisplayData, setDisplayData, setResourceHints, toString, validate, validate

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

DEFAULT_TIMEOUT

public static final Duration DEFAULT_TIMEOUT

The default Duration to wait until completion of user code. A UserCodeTimeoutException is thrown when Caller.call(RequestT), SetupTeardown.setup(), or SetupTeardown.teardown() exceed this timeout.

REPEATABLE_ERROR_TYPES

public static final java.util.Set<java.lang.Class<? extends UserCodeExecutionException>> REPEATABLE_ERROR_TYPES

Set of UserCodeExecutionExceptions that warrant repeating. Not all errors should be repeat execution such as bad or unauthenticated requests. However, certain errors such as timeouts, remote system or quota exceed errors may not be related to the code but due to the remote system and thus warrant repeating.

Method Detail

of

public static <RequestT,ResponseT> RequestResponseIO<RequestT,ResponseT> of(Caller<RequestT,ResponseT> caller,
                                                                            Coder<ResponseT> responseTCoder)

Instantiates a RequestResponseIO with a Caller and a ResponseT Coder. Checks for the Caller's SerializableUtils.ensureSerializable(T) serializable errors.

ofCallerAndSetupTeardown

public static <RequestT,ResponseT,CallerSetupTeardownT extends Caller<RequestT,ResponseT> & SetupTeardown> RequestResponseIO<RequestT,ResponseT> ofCallerAndSetupTeardown(CallerSetupTeardownT implementsCallerAndSetupTeardown,
                                                                                                                                                                          Coder<ResponseT> responseTCoder)

Instantiates a RequestResponseIO with a ResponseT Coder and an implementation of both the Caller and SetupTeardown interfaces. Checks SerializableUtils.ensureSerializable(T) serializable errors.

withTimeout

public RequestResponseIO<RequestT,ResponseT> withTimeout(Duration value)

Overrides the DEFAULT_TIMEOUT expected timeout of all user custom code. If user custom code exceeds this timeout, then a UserCodeTimeoutException is thrown. User custom code may throw this exception prior to the configured timeout value on their own.

withoutRepeater

public RequestResponseIO<RequestT,ResponseT> withoutRepeater()

Turns off repeat invocations (default is on) of SetupTeardown and Caller, using the Repeater, in the setting of REPEATABLE_ERROR_TYPES.

withCallShouldBackoff

public RequestResponseIO<RequestT,ResponseT> withCallShouldBackoff(CallShouldBackoff<ResponseT> value)

Overrides the private no-op implementation of CallShouldBackoff that determines whether the DoFn should hold RequestTs. Without this configuration, RequestTs are never held; no-op implemented CallShouldBackoff.isTrue() always returns false.

withCache

public RequestResponseIO<RequestT,ResponseT> withCache(Cache.Pair<RequestT,ResponseT> pair)

Configures RequestResponseIO for reading and writing RequestT and ResponseT pairs using a cache. RequestResponseIO, by default, does not interact with a cache.

When reading, the transform Flattens the ResponseT PCollection
 of successful pairs with that resulting from API calls of RequestTs of unsuccessful pairs.
 

When writing, the transform Flattens the ResponseT PCollection of
    successful pairs with that resulting from API calls of RequestTs.
 

withMonitoringConfiguration

public RequestResponseIO<RequestT,ResponseT> withMonitoringConfiguration(Monitoring value)

expand

public Result<ResponseT> expand(PCollection<RequestT> input)

Description copied from class: PTransform

Override this method to specify how this PTransform should be expanded on the given InputT.

NOTE: This method should not be called directly. Instead apply the PTransform should be applied to the InputT using the apply method.

Composite transforms, which are defined in terms of other transforms, should return the output of one of the composed transforms. Non-composite transforms, which do not apply any transforms internally, should return a new unbound output and register evaluators (via backend-specific registration methods).

Specified by:

expand in class PTransform<PCollection<RequestT>,Result<ResponseT>>