apache_beam.runners.direct.evaluation_context module

EvaluationContext tracks global state, triggers and watermarks.

class apache_beam.runners.direct.evaluation_context.EvaluationContext(pipeline_options, bundle_factory: BundleFactory, root_transforms, value_to_consumers, step_names, views: Iterable[apache_beam.pvalue.AsSideInput], clock)

Bases: object

Evaluation context with the global state information of the pipeline.

The evaluation context for a specific pipeline being executed by the DirectRunner. Contains state shared within the execution across all transforms.

EvaluationContext contains shared state for an execution of the DirectRunner that can be used while evaluating a PTransform. This consists of views into underlying state and watermark implementations, access to read and write side inputs, and constructing counter sets and execution contexts. This includes executing callbacks asynchronously when state changes to the appropriate point (e.g. when a side input is requested and known to be empty).

EvaluationContext also handles results by committing finalizing bundles based on the current global state and updating the global state appropriately. This includes updating the per-(step,key) state, updating global watermarks, and executing any callbacks that can be executed.

metrics()

is_root_transform(applied_ptransform: apache_beam.pipeline.AppliedPTransform) → bool

handle_result(completed_bundle: _Bundle, completed_timers, result: TransformResult)

Handle the provided result produced after evaluating the input bundle.

Handle the provided TransformResult, produced after evaluating the provided committed bundle (potentially None, if the result of a root PTransform).

The result is the output of running the transform contained in the TransformResult on the contents of the provided bundle.

Parameters:

• completed_bundle – the bundle that was processed to produce the result.
• completed_timers – the timers that were delivered to produce the completed_bundle.
• result – the TransformResult of evaluating the input bundle

Returns:

the committed bundles contained within the handled result.

get_aggregator_values(aggregator_or_name)

schedule_pending_unblocked_tasks(executor_service)

get_execution_context(applied_ptransform: apache_beam.pipeline.AppliedPTransform) → apache_beam.runners.direct.evaluation_context._ExecutionContext

create_bundle(output_pcollection: Union[apache_beam.pvalue.PBegin, apache_beam.pvalue.PCollection]) → _Bundle

Create an uncommitted bundle for the specified PCollection.

create_empty_committed_bundle(output_pcollection: apache_beam.pvalue.PCollection) → _Bundle

Create empty bundle useful for triggering evaluation.

extract_all_timers() → Tuple[List[Tuple[apache_beam.pipeline.AppliedPTransform, List[TimerFiring]]], bool]

is_done(transform: Optional[apache_beam.pipeline.AppliedPTransform] = None) → bool

Checks completion of a step or the pipeline.

Parameters:transform – AppliedPTransform to check for completion. Returns:True if the step will not produce additional output. If transform is None returns true if all steps are done.

get_value_or_block_until_ready(side_input, task, block_until)

shutdown()

class apache_beam.runners.direct.evaluation_context.DirectUnmergedState

Bases: apache_beam.transforms.trigger.InMemoryUnmergedState

UnmergedState implementation for the DirectRunner.

class apache_beam.runners.direct.evaluation_context.DirectStepContext(existing_keyed_state)

Bases: object

Context for the currently-executing step.

get_keyed_state(key)