Package org.apache.beam.runners.spark.io

Class MicrobatchSource<T,CheckpointMarkT extends UnboundedSource.CheckpointMark>

java.lang.Object

org.apache.beam.sdk.io.Source<T>

org.apache.beam.runners.spark.io.MicrobatchSource<T,CheckpointMarkT>

All Implemented Interfaces:

Serializable, HasDisplayData

public class MicrobatchSource<T,CheckpointMarkT extends UnboundedSource.CheckpointMark>
extends Source<T>

A Source that accommodates Spark's micro-batch oriented nature and wraps an UnboundedSource.

See Also:

• Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
class	MicrobatchSource.Reader	Mostly based on BoundedReadFromUnboundedSource's UnboundedToBoundedSourceAdapter, with some adjustments for Spark specifics.

Method Summary

Modifier and Type	Method	Description
static void	clearCache()
boolean	equals(@Nullable Object o)
Coder<CheckpointMarkT>	getCheckpointMarkCoder()
String	getId()
Source.Reader<T>	getOrCreateReader(PipelineOptions options, CheckpointMarkT checkpointMark)
Coder<T>	getOutputCoder()	Returns the Coder to use for the data read from this source.
int	hashCode()
void	validate()	Checks that this source is valid, before it can be used in a pipeline.

Methods inherited from class org.apache.beam.sdk.io.Source

getDefaultOutputCoder, populateDisplayData

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, toString, wait, wait, wait

Method Details

getOrCreateReader

public Source.Reader<T> getOrCreateReader(PipelineOptions options,
 CheckpointMarkT checkpointMark)
                                   throws IOException

Throws:

IOException

validate

public void validate()

Description copied from class: Source

Checks that this source is valid, before it can be used in a pipeline.

It is recommended to use Preconditions for implementing this method.

Overrides:

validate in class Source<T>

getOutputCoder

public Coder<T> getOutputCoder()

Description copied from class: Source

Returns the Coder to use for the data read from this source.

Overrides:

getOutputCoder in class Source<T>

getCheckpointMarkCoder

public Coder<CheckpointMarkT> getCheckpointMarkCoder()

getId

public String getId()

equals

public boolean equals(@Nullable Object o)

Overrides:

equals in class Object

hashCode

public int hashCode()

Overrides:

hashCode in class Object

clearCache

public static void clearCache()