apache_beam.runners.interactive.caching.reify module

Module for transforms that reifies and unreifies PCollection values with window info.

For internal use only; no backwards-compatibility guarantees.

class apache_beam.runners.interactive.caching.reify.Reify(*unused_args, **unused_kwargs)[source]

Bases: DoFn

Reifies elements with window info into windowed values.

Internally used to capture window info with each element into cache for replayability.

process(e, w=WindowParam, p=PaneInfoParam, t=TimestampParam)[source]
class apache_beam.runners.interactive.caching.reify.Unreify(*unused_args, **unused_kwargs)[source]

Bases: DoFn

Unreifies elements from windowed values.

Cached values are elements with window info. This unpacks the elements.

process(e)[source]
apache_beam.runners.interactive.caching.reify.reify_to_cache(pcoll: PCollection, cache_key: str, cache_manager: CacheManager, reify_label: str | None = None, write_cache_label: str | None = None, is_capture: bool = False) → PValue[source]

Reifies elements into windowed values and write to cache.

Parameters:

  • pcoll – The PCollection to be cached.

  • cache_key – The key of the cache.

  • cache_manager – The cache manager to manage the cache.

  • reify_label – (optional) A transform label for the Reify transform.

  • write_cache_label – (optional) A transform label for the cache-writing transform.

  • is_capture – Whether the cache is capturing a record of recordable sources.

apache_beam.runners.interactive.caching.reify.unreify_from_cache(pipeline: Pipeline, cache_key: str, cache_manager: CacheManager, element_type: type | None = None, source_label: str | None = None, unreify_label: str | None = None) → PCollection[source]

Reads from cache and unreifies elements from windowed values.

pipeline: The pipeline that’s reading from the cache. cache_key: The key of the cache. cache_manager: The cache manager to manage the cache. element_type: (optional) The element type of the PCollection’s elements. source_label: (optional) A transform label for the cache-reading transform. unreify_label: (optional) A transform label for the Unreify transform.