Package org.apache.beam.sdk.io

Class CountingSource.UnboundedCountingSource

java.lang.Object

org.apache.beam.sdk.io.Source<Long>

org.apache.beam.sdk.io.UnboundedSource<Long,CountingSource.CounterMark>

org.apache.beam.sdk.io.CountingSource.UnboundedCountingSource

All Implemented Interfaces:

Serializable, HasDisplayData

Enclosing class:

CountingSource

public static class CountingSource.UnboundedCountingSource
extends UnboundedSource<Long,CountingSource.CounterMark>

An implementation of CountingSource that produces an unbounded PCollection.

See Also:

• Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.beam.sdk.io.UnboundedSource

UnboundedSource.CheckpointMark, UnboundedSource.UnboundedReader<OutputT>

Nested classes/interfaces inherited from class org.apache.beam.sdk.io.Source

Source.Reader<T>

Method Summary

Modifier and Type	Method	Description
UnboundedSource.UnboundedReader<Long>	createReader(PipelineOptions options, CountingSource.CounterMark checkpointMark)	Create a new UnboundedSource.UnboundedReader to read from this source, resuming from the given checkpoint if present.
boolean	equals(@Nullable Object other)
Coder<CountingSource.CounterMark>	getCheckpointMarkCoder()	Returns a Coder for encoding and decoding the checkpoints for this source.
Coder<Long>	getOutputCoder()	Returns the Coder to use for the data read from this source.
int	hashCode()
List<? extends UnboundedSource<Long,CountingSource.CounterMark>>	split(int desiredNumSplits, PipelineOptions options)	Splits an unbounded source desiredNumSplits ways by giving each split every desiredNumSplitsth element that this CountingSource.UnboundedCountingSource produces.
CountingSource.UnboundedCountingSource	to(long end)	Returns an CountingSource.UnboundedCountingSource like this one but with the specified exclusive limit.
CountingSource.UnboundedCountingSource	withRate(long elementsPerPeriod, Duration period)	Returns an CountingSource.UnboundedCountingSource like this one with the specified period.
CountingSource.UnboundedCountingSource	withTimestampFn(SerializableFunction<Long,Instant> timestampFn)	Returns an CountingSource.UnboundedCountingSource like this one where the timestamp of output elements are supplied by the specified function.

Methods inherited from class org.apache.beam.sdk.io.UnboundedSource

offsetBasedDeduplicationSupported, requiresDeduping

Methods inherited from class org.apache.beam.sdk.io.Source

getDefaultOutputCoder, populateDisplayData, validate

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, toString, wait, wait, wait

Method Details

withRate

public CountingSource.UnboundedCountingSource withRate(long elementsPerPeriod,
 Duration period)

Returns an CountingSource.UnboundedCountingSource like this one with the specified period. Elements will be produced with an interval between them equal to the period.

withTimestampFn

public CountingSource.UnboundedCountingSource withTimestampFn(SerializableFunction<Long,Instant> timestampFn)

Returns an CountingSource.UnboundedCountingSource like this one where the timestamp of output elements are supplied by the specified function.

Note that timestamps produced by timestampFn may not decrease.

to

public CountingSource.UnboundedCountingSource to(long end)

Returns an CountingSource.UnboundedCountingSource like this one but with the specified exclusive limit.

split

public List<? extends UnboundedSource<Long,CountingSource.CounterMark>> split(int desiredNumSplits,
 PipelineOptions options)
                                                                       throws Exception

Splits an unbounded source desiredNumSplits ways by giving each split every desiredNumSplitsth element that this CountingSource.UnboundedCountingSource produces.

E.g., if a source produces all even numbers [0, 2, 4, 6, 8, ...) and we want to split into 3 new sources, then the new sources will produce numbers that are 6 apart and are offset at the start by the original stride: [0, 6, 12, ...), [2, 8, 14, ...), and [4, 10, 16, ...).

Specified by:

split in class UnboundedSource<Long,CountingSource.CounterMark>

Throws:

Exception

createReader

public UnboundedSource.UnboundedReader<Long> createReader(PipelineOptions options,
 CountingSource.CounterMark checkpointMark)

Description copied from class: UnboundedSource

Create a new UnboundedSource.UnboundedReader to read from this source, resuming from the given checkpoint if present.

Specified by:

createReader in class UnboundedSource<Long,CountingSource.CounterMark>

getCheckpointMarkCoder

public Coder<CountingSource.CounterMark> getCheckpointMarkCoder()

Description copied from class: UnboundedSource

Returns a Coder for encoding and decoding the checkpoints for this source.

Specified by:

getCheckpointMarkCoder in class UnboundedSource<Long,CountingSource.CounterMark>

getOutputCoder

public Coder<Long> getOutputCoder()

Description copied from class: Source

Returns the Coder to use for the data read from this source.

Overrides:

getOutputCoder in class Source<Long>

equals

public boolean equals(@Nullable Object other)

Overrides:

equals in class Object

hashCode

public int hashCode()

Overrides:

hashCode in class Object