Class RpcQosOptions.Builder

java.lang.Object
org.apache.beam.sdk.io.gcp.firestore.RpcQosOptions.Builder
Enclosing class:
RpcQosOptions

public static final class RpcQosOptions.Builder extends Object
Mutable Builder class for creating instances of RpcQosOptions.

A new instance of RpcQosOptions.Builder can be created via RpcQosOptions.newBuilder().

NOTE: All with methods in this class function as set rather than copy with new value

  • Method Details

    • withMaxAttempts

      public RpcQosOptions.Builder withMaxAttempts(int maxAttempts)
      Configure the maximum number of times a request will be attempted for a complete successful result.

      For a stream based response, the full stream read must complete within the specified number of attempts. Restarting a stream will count as a new attempt.

      Default Value: 5

      Parameters:
      maxAttempts - an int in the range 1 invalid input: '<'= maxAttempts invalid input: '<'= 5
      Returns:
      this builder
      See Also:
    • withInitialBackoff

      public RpcQosOptions.Builder withInitialBackoff(Duration initialBackoff)
      Configure the initial backoff duration to be used before retrying a request for the first time.

      Default Value: 5 sec

      Parameters:
      initialBackoff - a Duration in the range 5 sec invalid input: '<'= initialBackoff invalid input: '<'= 2 min
      Returns:
      this builder
      See Also:
    • withSamplePeriod

      public RpcQosOptions.Builder withSamplePeriod(Duration samplePeriod)
      Configure the length of time sampled request data will be retained.

      Default Value: 2 min

      Parameters:
      samplePeriod - a Duration in the range 2 min invalid input: '<'= samplePeriod invalid input: '<'= 20 min
      Returns:
      this builder
      See Also:
    • withSamplePeriodBucketSize

      public RpcQosOptions.Builder withSamplePeriodBucketSize(Duration samplePeriodBucketSize)
      Configure the size of buckets within the specified samplePeriod.

      Default Value: 10 sec

      Parameters:
      samplePeriodBucketSize - a Duration in the range 10 sec invalid input: '<'= samplePeriodBucketSize invalid input: '<'= 20 min
      Returns:
      this builder
      See Also:
    • withOverloadRatio

      public RpcQosOptions.Builder withOverloadRatio(double overloadRatio)
      The target ratio between requests sent and successful requests. This is "K" in the formula in SRE Book - Handling Overload

      Default Value: 1.05

      Parameters:
      overloadRatio - the target ratio between requests sent and successful requests. A double in the range 1.0 invalid input: '<'= overloadRatio invalid input: '<'= 1.5
      Returns:
      this builder
      See Also:
    • withThrottleDuration

      public RpcQosOptions.Builder withThrottleDuration(Duration throttleDuration)
      Configure the amount of time an attempt will be throttled if deemed necessary based on previous success rate.

      Default value: 5 sec

      Parameters:
      throttleDuration - a Duration in the range 5 sec invalid input: '<'= throttleDuration invalid input: '<'= 1 min
      Returns:
      this builder
      See Also:
    • withBatchInitialCount

      public RpcQosOptions.Builder withBatchInitialCount(int batchInitialCount)
      Configure the initial size of a batch; used in the absence of the QoS system having significant data to determine a better batch size.

      Default Value: 20

      Parameters:
      batchInitialCount - an int in the range 1 invalid input: '<'= batchInitialCount invalid input: '<'= 500
      Returns:
      this builder
      See Also:
    • withBatchMaxCount

      public RpcQosOptions.Builder withBatchMaxCount(int batchMaxCount)
      Configure the maximum number of writes to include in a batch. (Used in the absence of the QoS system having significant data to determine a better value). The actual number of writes per request may be lower if batchMaxBytes is reached first.

      Default Value: 500

      Parameters:
      batchMaxCount - an int in the range 1 invalid input: '<'= batchMaxCount invalid input: '<'= 500
      Returns:
      this builder
      See Also:
    • withBatchMaxBytes

      public RpcQosOptions.Builder withBatchMaxBytes(long batchMaxBytes)
      Configure the maximum number of bytes to include in a batch. (Used in the absence of the QoS system having significant data to determine a better value). The actual number of bytes per request may be lower if batchMaxCount is reached first.

      Default Value: 9.5 MiB

      Parameters:
      batchMaxBytes - an int in the range 1 B invalid input: '<'= batchMaxBytes invalid input: '<'= 9.5 MiB
      Returns:
      this builder
      See Also:
    • withBatchTargetLatency

      public RpcQosOptions.Builder withBatchTargetLatency(Duration batchTargetLatency)
      Target latency for batch requests. It aims for a target response time per RPC: the response times for previous RPCs and the number of writes contained in them, calculates a rolling average time-per-write, and chooses the number of writes for future requests to hit the target time.

      Default Value: 5 sec

      Parameters:
      batchTargetLatency - a Duration in the range 5 sec invalid input: '<'= batchTargetLatency invalid input: '<'= 2 min
      Returns:
      this builder
      See Also:
    • withHintMaxNumWorkers

      public RpcQosOptions.Builder withHintMaxNumWorkers(int hintMaxNumWorkers)
      Provide a hint to the QoS system for the intended max number of workers for a pipeline. The provided value can be used to try to scale calculations to values appropriate for the pipeline as a whole.

      If you are running your pipeline on Cloud Dataflow this parameter should be set to the same value as maxNumWorkers

      Default Value: 500

      Parameters:
      hintMaxNumWorkers - an int in the range 1 invalid input: '<'= hintMaxNumWorkers invalid input: '<'= Integer.MAX_VALUE
      Returns:
      this builder
      See Also:
    • withRampupThrottlingDisabled

      public RpcQosOptions.Builder withRampupThrottlingDisabled()
      Disable the 500/50/5 ramp-up throttling of writes.

      By default, each worker starts with a budget of 500 / hintMaxNumWorkers writes per second, which grows by 50% every 5 minutes. With the default hintMaxNumWorkers this is 1 write per second, which can severely limit the throughput of pipelines running on fewer workers. Disabling ramp-up throttling is appropriate when the target collection is already serving traffic, or its documents are written with well distributed IDs.

      Writes remain subject to adaptive throttling in response to failed requests.

      This mirrors DatastoreIO.v1().write().withRampupThrottlingDisabled().

      Returns:
      this builder
      See Also:
    • withReportDiagnosticMetrics

      public RpcQosOptions.Builder withReportDiagnosticMetrics()
      Whether additional diagnostic metrics should be reported for a Transform.

      If invoked on this builder, additional detailed diagnostic metrics will be reported for the RPC QoS subsystem.

      This parameter should be used with care as it will output ~40 additional custom metrics which will count toward any possible pipeline metrics limits (For example Dataflow Custom Metrics limits and Cloud Monitoring custom metrics quota)

      Returns:
      this builder
      See Also:
    • build

      public RpcQosOptions build()
      Create a new instance of RpcQosOptions from the current builder state.

      All provided values will be validated for non-nullness and that each value falls within its allowed range.

      Returns:
      a new instance of RpcQosOptions from the current builder state.
      Throws:
      NullPointerException - if any nullable value is null
      IllegalArgumentException - if any provided value does not fall within its allowed range