apache_beam.io.concat_source module

For internal use only; no backwards-compatibility guarantees.

Concat Source, which reads the union of several other sources.

class apache_beam.io.concat_source.ConcatSource(sources)[source]

Bases: BoundedSource

For internal use only; no backwards-compatibility guarantees.

A BoundedSource that can group a set of BoundedSources.

Primarily for internal use, use the apache_beam.Flatten transform to create the union of several reads.

property sources
estimate_size()[source]
split(desired_bundle_size=None, start_position=None, stop_position=None)[source]
get_range_tracker(start_position=None, stop_position=None)[source]
read(range_tracker)[source]
default_output_coder()[source]
class apache_beam.io.concat_source.ConcatRangeTracker(start, end, source_bundles)[source]

Bases: RangeTracker

For internal use only; no backwards-compatibility guarantees.

Range tracker for ConcatSource

Initializes ConcatRangeTracker

Parameters:
  • start – start position, a tuple of (source_index, source_position)

  • end – end position, a tuple of (source_index, source_position)

  • source_bundles – the list of source bundles in the ConcatSource

start_position()[source]
stop_position()[source]
try_claim(pos)[source]
try_split(pos)[source]
set_current_position(pos)[source]
position_at_fraction(fraction)[source]
fraction_consumed()[source]
local_to_global(source_ix, source_frac)[source]
global_to_local(frac)[source]
sub_range_tracker(source_ix)[source]