apache_beam.internal package

Submodules

apache_beam.internal.pickler module

Pickler for values, functions, and classes.

For internal use only. No backwards compatibility guarantees.

Pickles created by the pickling library contain non-ASCII characters, so we base64-encode the results so that we can put them in a JSON objects. The pickler is used to embed FlatMap callable objects into the workflow JSON description.

The pickler module should be used to pickle functions and modules; for values, the coders.*PickleCoder classes should be used instead.

apache_beam.internal.pickler.dump_session(file_path)[source]

For internal use only; no backwards-compatibility guarantees.

Pickle the current python session to be used in the worker.

Note: Due to the inconsistency in the first dump of dill dump_session we create and load the dump twice to have consistent results in the worker and the running session. Check: https://github.com/uqfoundation/dill/issues/195

apache_beam.internal.pickler.dumps(o, enable_trace=True)[source]

For internal use only; no backwards-compatibility guarantees.

apache_beam.internal.pickler.load_session(file_path)[source]
apache_beam.internal.pickler.loads(encoded, enable_trace=True)[source]

For internal use only; no backwards-compatibility guarantees.

apache_beam.internal.pickler.new_save_module_dict(pickler, obj)[source]
apache_beam.internal.pickler.save_module(pickler, obj)[source]

apache_beam.internal.util module

Utility functions used throughout the package.

For internal use only. No backwards compatibility guarantees.

class apache_beam.internal.util.ArgumentPlaceholder[source]

Bases: object

For internal use only; no backwards-compatibility guarantees.

A place holder object replacing PValues in argument lists.

A Fn object can take any number of “side inputs”, which are PValues that will be evaluated during pipeline execution and will be provided to the function at the moment of its execution as positional or keyword arguments.

This is used only internally and should never be used by user code. A custom Fn object by the time it executes will have such values replaced with real computed values.

apache_beam.internal.util.insert_values_in_args(args, kwargs, values)[source]

For internal use only; no backwards-compatibility guarantees.

Replaces all placeholders in args/kwargs with actual values.

Parameters:
  • args – A list of positional arguments.
  • kwargs – A dictionary of keyword arguments.
  • values – A list of values that will be used to replace placeholder values.
Returns:

A 2-tuple containing a modified list of positional arguments, and a modified dictionary of keyword arguments.
apache_beam.internal.util.remove_objects_from_args(args, kwargs, pvalue_classes)[source]

For internal use only; no backwards-compatibility guarantees.

Replaces all objects of a given type in args/kwargs with a placeholder.

Parameters:
  • args – A list of positional arguments.
  • kwargs – A dictionary of keyword arguments.
  • pvalue_classes – A tuple of class objects representing the types of the arguments that must be replaced with a placeholder value (instance of ArgumentPlaceholder)
Returns:

A 3-tuple containing a modified list of positional arguments, a modified dictionary of keyword arguments, and a list of all objects replaced with a placeholder value.
apache_beam.internal.util.run_using_threadpool(fn_to_execute, inputs, pool_size)[source]

For internal use only; no backwards-compatibility guarantees.

Runs the given function on given inputs using a thread pool.

Parameters:
  • fn_to_execute – Function to execute
  • inputs – Inputs on which given function will be executed in parallel.
  • pool_size – Size of thread pool.
Returns:

Results retrieved after executing the given function on given inputs.

Module contents

For internal use only; no backwards-compatibility guarantees.