apache_beam.runners.interactive.caching.reify module

Module for transforms that reifies and unreifies PCollection values with window info.

For internal use only; no backwards-compatibility guarantees.

class apache_beam.runners.interactive.caching.reify.Reify(*unused_args, **unused_kwargs)[source]

Bases: apache_beam.transforms.core.DoFn

Reifies elements with window info into windowed values.

Internally used to capture window info with each element into cache for replayability.

process(e, w=WindowParam, p=PaneInfoParam, t=TimestampParam)[source]
class apache_beam.runners.interactive.caching.reify.Unreify(*unused_args, **unused_kwargs)[source]

Bases: apache_beam.transforms.core.DoFn

Unreifies elements from windowed values.

Cached values are elements with window info. This unpacks the elements.

process(e)[source]
apache_beam.runners.interactive.caching.reify.reify_to_cache(pcoll: apache_beam.pvalue.PCollection, cache_key: str, cache_manager: apache_beam.runners.interactive.cache_manager.CacheManager, reify_label: Optional[str] = None, write_cache_label: Optional[str] = None, is_capture: bool = False) → apache_beam.pvalue.PValue[source]

Reifies elements into windowed values and write to cache.

Parameters:
  • pcoll – The PCollection to be cached.
  • cache_key – The key of the cache.
  • cache_manager – The cache manager to manage the cache.
  • reify_label – (optional) A transform label for the Reify transform.
  • write_cache_label – (optional) A transform label for the cache-writing transform.
  • is_capture – Whether the cache is capturing a record of recordable sources.
apache_beam.runners.interactive.caching.reify.unreify_from_cache(pipeline: apache_beam.pipeline.Pipeline, cache_key: str, cache_manager: apache_beam.runners.interactive.cache_manager.CacheManager, element_type: Optional[type] = None, source_label: Optional[str] = None, unreify_label: Optional[str] = None) → apache_beam.pvalue.PCollection[source]

Reads from cache and unreifies elements from windowed values.

pipeline: The pipeline that’s reading from the cache. cache_key: The key of the cache. cache_manager: The cache manager to manage the cache. element_type: (optional) The element type of the PCollection’s elements. source_label: (optional) A transform label for the cache-reading transform. unreify_label: (optional) A transform label for the Unreify transform.