apache_beam.io.gcp.datastore.v1new.helper module

Cloud Datastore client and test functions.

For internal use only; no backwards-compatibility guarantees.

apache_beam.io.gcp.datastore.v1new.helper.get_client(*args, **kwargs)

Returns a Cloud Datastore client.

apache_beam.io.gcp.datastore.v1new.helper.retry_on_rpc_error(exception)

A retry filter for Cloud Datastore RPCErrors.

apache_beam.io.gcp.datastore.v1new.helper.write_mutations(batch, throttler, rpc_stats_callback, throttle_delay=1)

A helper function to write a batch of mutations to Cloud Datastore.

If a commit fails, it will be retried up to 5 times. All mutations in the batch will be committed again, even if the commit was partially successful. If the retry limit is exceeded, the last exception from Cloud Datastore will be raised.

Assumes that the Datastore client library does not perform any retries on commits. It has not been determined how such retries would interact with the retries and throttler used here. See google.cloud.datastore_v1.gapic.datastore_client_config for retry config.

Parameters:

• batch – (Batch) An instance of an in-progress batch.
• rpc_stats_callback – a function to call with arguments successes and failures and throttled_secs; this is called to record successful and failed RPCs to Datastore and time spent waiting for throttling.
• throttler – (apache_beam.io.gcp.datastore.v1.adaptive_throttler. AdaptiveThrottler) Throttler instance used to select requests to be throttled.
• throttle_delay – (float) time in seconds to sleep when throttled.

Returns:

(int) The latency of the successful RPC in milliseconds.

apache_beam.io.gcp.datastore.v1new.helper.create_entities(count, id_or_name=False)

Creates a list of entities with random keys.

apache_beam.io.gcp.datastore.v1new.helper.create_client_entities(count, id_or_name=False)

Creates a list of client-style entities with random keys.