org.apache.beam.sdk.io.aws2.common

Class AsyncBatchWriteHandler<RecT,ResT>

java.lang.Object

org.apache.beam.sdk.io.aws2.common.AsyncBatchWriteHandler<RecT,ResT>

Type Parameters:

RecT - Record type in batch

ResT - Potentially erroneous result that needs to be correlated to a record using failedRecords(List, List)

@NotThreadSafe
 @Internal
public abstract class AsyncBatchWriteHandler<RecT,ResT>
extends java.lang.Object

Async handler that automatically retries unprocessed records in case of a partial success.

The handler enforces the provided upper limit of concurrent requests. Once that limit is reached any further call to batchWrite(String, List) will block until another request completed.

The handler is fail fast and won't submit any further request after a failure. Async failures can be polled using checkForAsyncFailure().

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static interface	AsyncBatchWriteHandler.Stats Statistics on the batch request.

Field Summary

Fields

Modifier and Type	Field and Description
protected java.util.function.Function<ResT,java.lang.String>	errorCodeFn
protected java.util.function.BiFunction<java.lang.String,java.util.List<RecT>,java.util.concurrent.CompletableFuture<java.util.List<ResT>>>	submitFn

Constructor Summary

Constructors

Modifier	Constructor and Description
protected	AsyncBatchWriteHandler(int concurrency, org.apache.beam.sdk.util.FluentBackoff backoff, AsyncBatchWriteHandler.Stats stats, java.util.function.Function<ResT,java.lang.String> errorCodeFn, java.util.function.BiFunction<java.lang.String,java.util.List<RecT>,java.util.concurrent.CompletableFuture<java.util.List<ResT>>> submitFn)

Method Summary

Modifier and Type	Method and Description
void	batchWrite(java.lang.String destination, java.util.List<RecT> records) Asynchronously trigger a batch write request (unless already in error state).
void	batchWrite(java.lang.String destination, java.util.List<RecT> records, boolean throwAsyncFailures) Asynchronously trigger a batch write request (unless already in error state).
static <RecT,ErrT> AsyncBatchWriteHandler<RecT,ErrT>	byId(int concurrency, org.apache.beam.sdk.util.FluentBackoff backoff, AsyncBatchWriteHandler.Stats stats, java.util.function.BiFunction<java.lang.String,java.util.List<RecT>,java.util.concurrent.CompletableFuture<java.util.List<ErrT>>> submitFn, java.util.function.Function<ErrT,java.lang.String> errorCodeFn, java.util.function.Function<RecT,java.lang.String> recordIdFn, java.util.function.Function<ErrT,java.lang.String> errorIdFn) AsyncBatchWriteHandler that correlates records and results by id, all results are erroneous.
static <RecT,ErrT> AsyncBatchWriteHandler<RecT,ErrT>	byId(int concurrency, int partialRetries, RetryConfiguration retry, AsyncBatchWriteHandler.Stats stats, java.util.function.BiFunction<java.lang.String,java.util.List<RecT>,java.util.concurrent.CompletableFuture<java.util.List<ErrT>>> submitFn, java.util.function.Function<ErrT,java.lang.String> errorCodeFn, java.util.function.Function<RecT,java.lang.String> recordIdFn, java.util.function.Function<ErrT,java.lang.String> errorIdFn) AsyncBatchWriteHandler that correlates records and results by id, all results are erroneous.
static <RecT,ResT> AsyncBatchWriteHandler<RecT,ResT>	byPosition(int concurrency, org.apache.beam.sdk.util.FluentBackoff backoff, AsyncBatchWriteHandler.Stats stats, java.util.function.BiFunction<java.lang.String,java.util.List<RecT>,java.util.concurrent.CompletableFuture<java.util.List<ResT>>> submitFn, java.util.function.Function<ResT,java.lang.String> errorCodeFn) AsyncBatchWriteHandler that correlates records and results by position in the respective list.
static <RecT,ResT> AsyncBatchWriteHandler<RecT,ResT>	byPosition(int concurrency, int partialRetries, RetryConfiguration retry, AsyncBatchWriteHandler.Stats stats, java.util.function.BiFunction<java.lang.String,java.util.List<RecT>,java.util.concurrent.CompletableFuture<java.util.List<ResT>>> submitFn, java.util.function.Function<ResT,java.lang.String> errorCodeFn) AsyncBatchWriteHandler that correlates records and results by position in the respective list.
void	checkForAsyncFailure() Check if any failure happened async.
protected abstract java.util.List<RecT>	failedRecords(java.util.List<RecT> records, java.util.List<ResT> results)
boolean	hasErrored() If this handler has errored since it was last reset.
protected abstract boolean	hasFailedRecords(java.util.List<ResT> results)
int	requestsInProgress()
void	reset()
void	waitForCompletion() Wait for all pending requests to complete and check for failures.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

submitFn

protected final java.util.function.BiFunction<java.lang.String,java.util.List<RecT>,java.util.concurrent.CompletableFuture<java.util.List<ResT>>> submitFn

errorCodeFn

protected final java.util.function.Function<ResT,java.lang.String> errorCodeFn

Constructor Detail

AsyncBatchWriteHandler

protected AsyncBatchWriteHandler(int concurrency,
                                 org.apache.beam.sdk.util.FluentBackoff backoff,
                                 AsyncBatchWriteHandler.Stats stats,
                                 java.util.function.Function<ResT,java.lang.String> errorCodeFn,
                                 java.util.function.BiFunction<java.lang.String,java.util.List<RecT>,java.util.concurrent.CompletableFuture<java.util.List<ResT>>> submitFn)

Method Detail

requestsInProgress

public final int requestsInProgress()

reset

public final void reset()

hasErrored

public final boolean hasErrored()

If this handler has errored since it was last reset.

checkForAsyncFailure

public final void checkForAsyncFailure()
                                throws java.lang.Throwable

Check if any failure happened async.

Throws:

java.lang.Throwable - The last async failure, afterwards reset it.

waitForCompletion

public final void waitForCompletion()
                             throws java.lang.Throwable

Wait for all pending requests to complete and check for failures.

Throws:

java.lang.Throwable - The last async failure if present using checkForAsyncFailure()

batchWrite

public final void batchWrite(java.lang.String destination,
                             java.util.List<RecT> records)
                      throws java.lang.Throwable

Asynchronously trigger a batch write request (unless already in error state).

This will respect the concurrency limit of the handler and first wait for a permit.

Throws:

java.lang.Throwable - The last async failure if present using checkForAsyncFailure()

batchWrite

public final void batchWrite(java.lang.String destination,
                             java.util.List<RecT> records,
                             boolean throwAsyncFailures)
                      throws java.lang.Throwable

Asynchronously trigger a batch write request (unless already in error state).

This will respect the concurrency limit of the handler and first wait for a permit.

Parameters:

throwAsyncFailures - If to check and throw pending async failures

Throws:

java.lang.Throwable - The last async failure if present using checkForAsyncFailure()

failedRecords

protected abstract java.util.List<RecT> failedRecords(java.util.List<RecT> records,
                                                      java.util.List<ResT> results)

hasFailedRecords

protected abstract boolean hasFailedRecords(java.util.List<ResT> results)

byPosition

public static <RecT,ResT> AsyncBatchWriteHandler<RecT,ResT> byPosition(int concurrency,
                                                                       int partialRetries,
                                                                       @Nullable
                                                                       RetryConfiguration retry,
                                                                       AsyncBatchWriteHandler.Stats stats,
                                                                       java.util.function.BiFunction<java.lang.String,java.util.List<RecT>,java.util.concurrent.CompletableFuture<java.util.List<ResT>>> submitFn,
                                                                       java.util.function.Function<ResT,java.lang.String> errorCodeFn)

AsyncBatchWriteHandler that correlates records and results by position in the respective list.

byPosition

public static <RecT,ResT> AsyncBatchWriteHandler<RecT,ResT> byPosition(int concurrency,
                                                                       org.apache.beam.sdk.util.FluentBackoff backoff,
                                                                       AsyncBatchWriteHandler.Stats stats,
                                                                       java.util.function.BiFunction<java.lang.String,java.util.List<RecT>,java.util.concurrent.CompletableFuture<java.util.List<ResT>>> submitFn,
                                                                       java.util.function.Function<ResT,java.lang.String> errorCodeFn)

AsyncBatchWriteHandler that correlates records and results by position in the respective list.

byId

public static <RecT,ErrT> AsyncBatchWriteHandler<RecT,ErrT> byId(int concurrency,
                                                                 int partialRetries,
                                                                 @Nullable
                                                                 RetryConfiguration retry,
                                                                 AsyncBatchWriteHandler.Stats stats,
                                                                 java.util.function.BiFunction<java.lang.String,java.util.List<RecT>,java.util.concurrent.CompletableFuture<java.util.List<ErrT>>> submitFn,
                                                                 java.util.function.Function<ErrT,java.lang.String> errorCodeFn,
                                                                 java.util.function.Function<RecT,java.lang.String> recordIdFn,
                                                                 java.util.function.Function<ErrT,java.lang.String> errorIdFn)

AsyncBatchWriteHandler that correlates records and results by id, all results are erroneous.

byId

public static <RecT,ErrT> AsyncBatchWriteHandler<RecT,ErrT> byId(int concurrency,
                                                                 org.apache.beam.sdk.util.FluentBackoff backoff,
                                                                 AsyncBatchWriteHandler.Stats stats,
                                                                 java.util.function.BiFunction<java.lang.String,java.util.List<RecT>,java.util.concurrent.CompletableFuture<java.util.List<ErrT>>> submitFn,
                                                                 java.util.function.Function<ErrT,java.lang.String> errorCodeFn,
                                                                 java.util.function.Function<RecT,java.lang.String> recordIdFn,
                                                                 java.util.function.Function<ErrT,java.lang.String> errorIdFn)

AsyncBatchWriteHandler that correlates records and results by id, all results are erroneous.