apache_beam.io.components.rate_limiter module

Rate Limiter classes for controlling access to external resources.

class apache_beam.io.components.rate_limiter.RateLimiter(namespace: str = '')

Bases: ABC

Abstract base class for RateLimiters.

abstract allow(**kwargs) → bool

Applies rate limiting to the request.

This method checks if the request is permitted by the rate limiting policy. Depending on the implementation and configuration, it may block (sleep) until the request is allowed, or return false if the rate limit retry is exceeded.

Parameters:

**kwargs – Keyword arguments specific to the RateLimiter implementation.

Returns:

True if the request is allowed, False if retries exceeded.

Return type:

bool

Raises:

Exception – If an underlying infrastructure error occurs (e.g. RPC failure).

class apache_beam.io.components.rate_limiter.EnvoyRateLimiter(service_address: str, domain: str, descriptors: List[Dict[str, str]], timeout: float = 5.0, block_until_allowed: bool = True, retries: int = 3, namespace: str = '')

Bases: RateLimiter

Rate limiter implementation that uses an external Envoy Rate Limit Service.

This limiter connects to a gRPC Envoy Rate Limit Service (RLS) to determine whether a request should be allowed. It supports defining a domain and a list of descriptors that correspond to the rate limit configuration in the RLS.

Parameters:

• service_address – Address of the Envoy RLS (e.g., ‘localhost:8081’).

• domain – The rate limit domain.

• descriptors – List of descriptors (key-value pairs).

• retries – Number of retries to attempt if rate limited, respected only if block_until_allowed is False.

• timeout – gRPC timeout in seconds.

• block_until_allowed – If enabled blocks until RateLimiter gets the token.

• namespace – the namespace to use for logging and signaling throttling is occurring.

class RateLimitServiceStub(channel)

Bases: object

Wrapper for gRPC stub to be compatible with envoy_data_plane messages.

The envoy-data-plane package uses ‘betterproto’ which generates async stubs for ‘grpclib’. As Beam uses standard synchronous ‘grpcio’, RateLimitServiceStub is a bridge class to use the betterproto Message types (RateLimitRequest) with a standard grpcio Channel.

init_connection()

allow(hits_added: int = 1) → bool

Calls the Envoy RLS to apply rate limits.

Sends a rate limit request to the configured Envoy Rate Limit Service. If ‘block_until_allowed’ is True, this method will sleep and retry if the limit is exceeded, effectively blocking until the request is permitted.

If ‘block_until_allowed’ is False, it will return False after the retry limit is exceeded.

Parameters:

hits_added – Number of hits to add to the rate limit.

Returns:

True if the request is allowed, False if retries exceeded.

Return type:

bool