apache_beam.runners.trivial_runner module

class apache_beam.runners.trivial_runner.TrivialRunner

Bases: apache_beam.runners.runner.PipelineRunner

A bare-bones batch Python pipeline runner illistrating how to use the RunnerAPI and FnAPI to execute pipelines.

Note that this runner is primarily for pedagogical purposes and is missing several features in order to keep it as simple as possible. Where possible pointers are provided which this should serve as a useful starting point.

run_portable_pipeline(pipeline, options)

execute_transform(transform_id, execution_state)

Execute a single transform.

execute_executable_stage(transform_proto, execution_state)

group_by_key_and_window(input_pcoll, output_pcoll, execution_state)

Groups the elements of input_pcoll, placing their output in output_pcoll.

supported_requirements() → Iterable[str]

class apache_beam.runners.trivial_runner.ExecutionState(optimized_pipeline)

Bases: object

A helper class holding various values and context during execution.

register_process_bundle_descriptor(process_bundle_descriptor: org.apache.beam.model.fn_execution.v1.beam_fn_api_pb2.ProcessBundleDescriptor)

get_pcollection_contents(pcoll_id: str) → List[bytes]

set_pcollection_contents(pcoll_id: str, chunks: List[bytes])

new_id(prefix='') → str

windowed_coder(pcollection_id: str) → apache_beam.coders.coders.Coder

windowing_strategy(pcollection_id: str) → apache_beam.transforms.core.Windowing

windowed_coder_id(pcollection_id: str) → str

apache_beam.runners.trivial_runner.is_primitive_transform(transform: org.apache.beam.model.pipeline.v1.beam_runner_api_pb2.PTransform) → bool

apache_beam.runners.trivial_runner.only_element(iterable: Iterable[T]) → T

apache_beam.runners.trivial_runner.decode_all(encoded_elements: bytes, coder: apache_beam.coders.coders.Coder) → Iterator[Any]

apache_beam.runners.trivial_runner.encode_all(elements: Iterator[T], coder: apache_beam.coders.coders.Coder) → bytes