apache_beam.io.concat_source module

For internal use only; no backwards-compatibility guarantees.

Concat Source, which reads the union of several other sources.

class apache_beam.io.concat_source.ConcatSource(sources)

Bases: apache_beam.io.iobase.BoundedSource

For internal use only; no backwards-compatibility guarantees.

A BoundedSource that can group a set of BoundedSources.

Primarily for internal use, use the apache_beam.Flatten transform to create the union of several reads.

sources

estimate_size()

split(desired_bundle_size=None, start_position=None, stop_position=None)

get_range_tracker(start_position=None, stop_position=None)

read(range_tracker)

default_output_coder()

class apache_beam.io.concat_source.ConcatRangeTracker(start, end, source_bundles)

Bases: apache_beam.io.iobase.RangeTracker

For internal use only; no backwards-compatibility guarantees.

Range tracker for ConcatSource

Initializes ConcatRangeTracker

Parameters:

• start – start position, a tuple of (source_index, source_position)
• end – end position, a tuple of (source_index, source_position)
• source_bundles – the list of source bundles in the ConcatSource

start_position()

stop_position()

try_claim(pos)

try_split(pos)

set_current_position(pos)

position_at_fraction(fraction)

fraction_consumed()

local_to_global(source_ix, source_frac)

global_to_local(frac)

sub_range_tracker(source_ix)