org.apache.beam.sdk.io.gcp.datastore

Class RampupThrottlingFn<T>

java.lang.Object

org.apache.beam.sdk.transforms.DoFn<T,T>

org.apache.beam.sdk.io.gcp.datastore.RampupThrottlingFn<T>

All Implemented Interfaces:

java.io.Serializable, HasDisplayData

public class RampupThrottlingFn<T>
extends DoFn<T,T>
implements java.io.Serializable

An implementation of a client-side throttler that enforces a gradual ramp-up, broadly in line with Datastore best practices. See also https://cloud.google.com/datastore/docs/best-practices#ramping_up_traffic.

See Also:

Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.beam.sdk.transforms.DoFn

DoFn.AlwaysFetched, DoFn.BoundedPerElement, DoFn.BundleFinalizer, DoFn.Element, DoFn.FieldAccess, DoFn.FinishBundle, DoFn.FinishBundleContext, DoFn.GetInitialRestriction, DoFn.GetInitialWatermarkEstimatorState, DoFn.GetRestrictionCoder, DoFn.GetSize, DoFn.GetWatermarkEstimatorStateCoder, DoFn.Key, DoFn.MultiOutputReceiver, DoFn.NewTracker, DoFn.NewWatermarkEstimator, DoFn.OnTimer, DoFn.OnTimerContext, DoFn.OnTimerFamily, DoFn.OnWindowExpiration, DoFn.OnWindowExpirationContext, DoFn.OutputReceiver<T>, DoFn.ProcessContext, DoFn.ProcessContinuation, DoFn.ProcessElement, DoFn.RequiresStableInput, DoFn.RequiresTimeSortedInput, DoFn.Restriction, DoFn.Setup, DoFn.SideInput, DoFn.SplitRestriction, DoFn.StartBundle, DoFn.StartBundleContext, DoFn.StateId, DoFn.Teardown, DoFn.TimerFamily, DoFn.TimerId, DoFn.Timestamp, DoFn.TruncateRestriction, DoFn.UnboundedPerElement, DoFn.WatermarkEstimatorState, DoFn.WindowedContext

Constructor Summary

Constructors

Constructor and Description
RampupThrottlingFn(int numWorkers, PCollectionView<Instant> timestampSideInput)
RampupThrottlingFn(ValueProvider<java.lang.Integer> numWorkers, PCollectionView<Instant> firstInstantSideInput)

Method Summary

Modifier and Type	Method and Description
void	populateDisplayData(DisplayData.Builder builder) Register display data for the given transform or component.
void	processElement(DoFn.ProcessContext c) Emit only as many elements as the exponentially increasing budget allows.
void	setup()

Methods inherited from class org.apache.beam.sdk.transforms.DoFn

getAllowedTimestampSkew, getInputTypeDescriptor, getOutputTypeDescriptor, prepareForProcessing

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

RampupThrottlingFn

public RampupThrottlingFn(ValueProvider<java.lang.Integer> numWorkers,
                          PCollectionView<Instant> firstInstantSideInput)

RampupThrottlingFn

public RampupThrottlingFn(int numWorkers,
                          PCollectionView<Instant> timestampSideInput)

Method Detail

setup

@DoFn.Setup
public void setup()

processElement

@DoFn.ProcessElement
public void processElement(DoFn.ProcessContext c)
                                         throws java.io.IOException,
                                                java.lang.InterruptedException

Emit only as many elements as the exponentially increasing budget allows.

Throws:

java.io.IOException

java.lang.InterruptedException

populateDisplayData

public void populateDisplayData(DisplayData.Builder builder)

Description copied from class: DoFn

Register display data for the given transform or component.

populateDisplayData(DisplayData.Builder) is invoked by Pipeline runners to collect display data via DisplayData.from(HasDisplayData). Implementations may call super.populateDisplayData(builder) in order to register display data in the current namespace, but should otherwise use subcomponent.populateDisplayData(builder) to use the namespace of the subcomponent.

By default, does not register any display data. Implementors may override this method to provide their own display data.

Specified by:

populateDisplayData in interface HasDisplayData

Overrides:

populateDisplayData in class DoFn<T,T>

Parameters:

builder - The builder to populate with display data.

See Also:

HasDisplayData