apache_beam.runners.interactive.cache_manager module

class apache_beam.runners.interactive.cache_manager.CacheManager

Bases: object

Abstract class for caching PCollections.

A PCollection cache is identified by labels, which consist of a prefix (either ‘full’ or ‘sample’) and a cache_label which is a hash of the PCollection derivation.

exists(*labels)

Returns if the PCollection cache exists.

is_latest_version(version, *labels)

Returns if the given version number is the latest.

read(*labels, **args)

Return the PCollection as a list as well as the version number.

Parameters:

• *labels – List of labels for PCollection instance.
• **args – Dict of additional arguments. Currently only ‘tail’ as a boolean. When tail is True, will wait and read new elements until the cache is complete.

Returns:

A tuple containing an iterator for the items in the PCollection and the

version number.

It is possible that the version numbers from read() and_latest_version() are different. This usually means that the cache’s been evicted (thus unavailable => read() returns version = -1), but it had reached version n before eviction.

write(value, *labels)

Writes the value to the given cache.

Parameters:

• value – An encodable (with corresponding PCoder) value
• *labels – List of labels for PCollection instance

clear(*labels)

Clears the cache entry of the given labels and returns True on success.

Parameters:

• value – An encodable (with corresponding PCoder) value
• *labels – List of labels for PCollection instance

source(*labels)

Returns a PTransform that reads the PCollection cache.

sink(labels, is_capture=False)

Returns a PTransform that writes the PCollection cache.

TODO(BEAM-10514): Make sure labels will not be converted into an arbitrarily long file path: e.g., windows has a 260 path limit.

save_pcoder(pcoder, *labels)

Saves pcoder for given PCollection.

Correct reading of PCollection from Cache requires PCoder to be known. This method saves desired PCoder for PCollection that will subsequently be used by sink(…), source(…), and, most importantly, read(…) method. The latter must be able to read a PCollection written by Beam using non-Beam IO.

Parameters:

• pcoder – A PCoder to be used for reading and writing a PCollection.
• *labels – List of labels for PCollection instance.

load_pcoder(*labels)

Returns previously saved PCoder for reading and writing PCollection.

cleanup()

Cleans up all the PCollection caches.

size(*labels)

Returns the size of the PCollection on disk in bytes.

class apache_beam.runners.interactive.cache_manager.FileBasedCacheManager(cache_dir=None, cache_format='text')

Bases: apache_beam.runners.interactive.cache_manager.CacheManager

Maps PCollections to local temp files for materialization.

size(*labels)

exists(*labels)

save_pcoder(pcoder, *labels)

load_pcoder(*labels)

read(*labels, **args)

write(values, *labels)

Imitates how a WriteCache transform works without running a pipeline.

For testing and cache manager development, not for production usage because the write is not sharded and does not use Beam execution model.

clear(*labels)

source(*labels)

sink(labels, is_capture=False)

cleanup()

class apache_beam.runners.interactive.cache_manager.ReadCache(cache_manager, label)

Bases: apache_beam.transforms.ptransform.PTransform

A PTransform that reads the PCollections from the cache.

expand(pbegin)

class apache_beam.runners.interactive.cache_manager.WriteCache(cache_manager, label, sample=False, sample_size=0, is_capture=False)

Bases: apache_beam.transforms.ptransform.PTransform

A PTransform that writes the PCollections to the cache.

expand(pcoll)

class apache_beam.runners.interactive.cache_manager.SafeFastPrimitivesCoder

Bases: apache_beam.coders.coders.Coder

This class add an quote/unquote step to escape special characters.

encode(value)

decode(value)