apache_beam.runners.interactive.recording_manager module

class apache_beam.runners.interactive.recording_manager.ElementStream(pcoll, var, cache_key, max_n, max_duration_secs)

Bases: object

A stream of elements from a given PCollection.

var

Returns the variable named that defined this PCollection.

pcoll

Returns the PCollection that supplies this stream with data.

cache_key

Returns the cache key for this stream.

display_id(suffix)

Returns a unique id able to be displayed in a web browser.

is_computed()

Returns True if no more elements will be recorded.

is_done()

Returns True if no more new elements will be yielded.

read(tail=True)

Reads the elements currently recorded.

class apache_beam.runners.interactive.recording_manager.Recording(user_pipeline, pcolls, result, max_n, max_duration_secs)

Bases: object

A group of PCollections from a given pipeline run.

is_computed()

Returns True if all PCollections are computed.

stream(pcoll)

Returns an ElementStream for a given PCollection.

computed()

Returns all computed ElementStreams.

uncomputed()

Returns all uncomputed ElementStreams.

cancel()

Cancels the recording.

wait_until_finish()

Waits until the pipeline is done and returns the final state.

This also marks any PCollections as computed right away if the pipeline is successful.

describe()

Returns a dictionary describing the cache and recording.

class apache_beam.runners.interactive.recording_manager.RecordingManager(user_pipeline, pipeline_var=None, test_limiters=None)

Bases: object

Manages recordings of PCollections for a given pipeline.

clear()

Clears all cached PCollections for this RecordingManager.

cancel()

Cancels the current background recording job.

describe()

Returns a dictionary describing the cache and recording.

record_pipeline()

Starts a background caching job for this RecordingManager’s pipeline.

record(pcolls, max_n, max_duration)

Records the given PCollections.

read(pcoll_name, pcoll, max_n, max_duration_secs)

Reads an ElementStream of a computed PCollection.

Returns None if an error occurs. The caller is responsible of validating if the given pcoll_name and pcoll can identify a watched and computed PCollection without ambiguity in the notebook.