apache_beam.runners.interactive.recording_manager module

class apache_beam.runners.interactive.recording_manager.ElementStream(pcoll: apache_beam.pvalue.PCollection, var: str, cache_key: str, max_n: int, max_duration_secs: float)

Bases: object

A stream of elements from a given PCollection.

var

Returns the variable named that defined this PCollection.

pcoll

Returns the PCollection that supplies this stream with data.

cache_key

Returns the cache key for this stream.

display_id(suffix: str) → str

Returns a unique id able to be displayed in a web browser.

is_computed() → bool

Returns True if no more elements will be recorded.

is_done() → bool

Returns True if no more new elements will be yielded.

read(tail: bool = True) → Any

Reads the elements currently recorded.

class apache_beam.runners.interactive.recording_manager.Recording(user_pipeline: apache_beam.pipeline.Pipeline, pcolls: List[apache_beam.pvalue.PCollection], result: beam.runner.PipelineResult, max_n: int, max_duration_secs: float)

Bases: object

A group of PCollections from a given pipeline run.

is_computed() → bool

Returns True if all PCollections are computed.

stream(pcoll: apache_beam.pvalue.PCollection) → apache_beam.runners.interactive.recording_manager.ElementStream

Returns an ElementStream for a given PCollection.

computed() → None

Returns all computed ElementStreams.

uncomputed() → None

Returns all uncomputed ElementStreams.

cancel() → None

Cancels the recording.

wait_until_finish() → None

Waits until the pipeline is done and returns the final state.

This also marks any PCollections as computed right away if the pipeline is successful.

describe() → Dict[str, int]

Returns a dictionary describing the cache and recording.

class apache_beam.runners.interactive.recording_manager.RecordingManager(user_pipeline: apache_beam.pipeline.Pipeline, pipeline_var: str = None, test_limiters: List[Limiter] = None)

Bases: object

Manages recordings of PCollections for a given pipeline.

clear() → None

Clears all cached PCollections for this RecordingManager.

cancel() → None

Cancels the current background recording job.

describe() → Dict[str, int]

Returns a dictionary describing the cache and recording.

record_pipeline() → bool

Starts a background caching job for this RecordingManager’s pipeline.

record(pcolls: List[apache_beam.pvalue.PCollection], max_n: int, max_duration: Union[int, str]) → apache_beam.runners.interactive.recording_manager.Recording

Records the given PCollections.

read(pcoll_name: str, pcoll: apache_beam.pvalue.PValue, max_n: int, max_duration_secs: float) → Union[None, apache_beam.runners.interactive.recording_manager.ElementStream]

Reads an ElementStream of a computed PCollection.

Returns None if an error occurs. The caller is responsible of validating if the given pcoll_name and pcoll can identify a watched and computed PCollection without ambiguity in the notebook.