apache_beam.runners.trivial_runner module

class apache_beam.runners.trivial_runner.TrivialRunner[source]

Bases: PipelineRunner

A bare-bones batch Python pipeline runner illistrating how to use the RunnerAPI and FnAPI to execute pipelines.

Note that this runner is primarily for pedagogical purposes and is missing several features in order to keep it as simple as possible. Where possible pointers are provided which this should serve as a useful starting point.

run_portable_pipeline(pipeline, options)[source]

execute_transform(transform_id, execution_state)[source]: Execute a single transform.

execute_executable_stage(transform_proto, execution_state)[source]

group_by_key_and_window(input_pcoll, output_pcoll, execution_state)[source]: Groups the elements of input_pcoll, placing their output in output_pcoll.

supported_requirements() → Iterable[str][source]

class apache_beam.runners.trivial_runner.ExecutionState(optimized_pipeline)[source]

Bases: object

A helper class holding various values and context during execution.

register_process_bundle_descriptor(process_bundle_descriptor: ProcessBundleDescriptor)[source]

get_pcollection_contents(pcoll_id: str) → List[bytes][source]

set_pcollection_contents(pcoll_id: str, chunks: List[bytes])[source]

new_id(prefix='') → str[source]

windowed_coder(pcollection_id: str) → Coder[source]

windowing_strategy(pcollection_id: str) → Windowing[source]

windowed_coder_id(pcollection_id: str) → str[source]

apache_beam.runners.trivial_runner.is_primitive_transform(transform: PTransform) → bool[source]

apache_beam.runners.trivial_runner.only_element(iterable: Iterable[T]) → T[source]

apache_beam.runners.trivial_runner.decode_all(encoded_elements: bytes, coder: Coder) → Iterator[Any][source]

apache_beam.runners.trivial_runner.encode_all(elements: Iterator[T], coder: Coder) → bytes[source]