org.apache.beam.runners.spark.io

Class MicrobatchSource<T,CheckpointMarkT extends UnboundedSource.CheckpointMark>

java.lang.Object

org.apache.beam.sdk.io.Source<T>

org.apache.beam.runners.spark.io.MicrobatchSource<T,CheckpointMarkT>

All Implemented Interfaces:

java.io.Serializable, HasDisplayData

public class MicrobatchSource<T,CheckpointMarkT extends UnboundedSource.CheckpointMark>
extends Source<T>

A Source that accommodates Spark's micro-batch oriented nature and wraps an UnboundedSource.

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
class	MicrobatchSource.Reader Mostly based on BoundedReadFromUnboundedSource's UnboundedToBoundedSourceAdapter, with some adjustments for Spark specifics.

Method Summary

Modifier and Type	Method and Description
static void	clearCache()
boolean	equals(java.lang.Object o)
Coder<CheckpointMarkT>	getCheckpointMarkCoder()
java.lang.String	getId()
Source.Reader<T>	getOrCreateReader(PipelineOptions options, CheckpointMarkT checkpointMark)
Coder<T>	getOutputCoder() Returns the Coder to use for the data read from this source.
int	hashCode()
void	validate() Checks that this source is valid, before it can be used in a pipeline.

Methods inherited from class org.apache.beam.sdk.io.Source

getDefaultOutputCoder, populateDisplayData

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, toString, wait, wait, wait

Method Detail

getOrCreateReader

public Source.Reader<T> getOrCreateReader(PipelineOptions options,
                                          CheckpointMarkT checkpointMark)
                                   throws java.io.IOException

Throws:

java.io.IOException

validate

public void validate()

Description copied from class: Source

Checks that this source is valid, before it can be used in a pipeline.

It is recommended to use Preconditions for implementing this method.

Overrides:

validate in class Source<T>

getOutputCoder

public Coder<T> getOutputCoder()

Description copied from class: Source

Returns the Coder to use for the data read from this source.

Overrides:

getOutputCoder in class Source<T>

getCheckpointMarkCoder

public Coder<CheckpointMarkT> getCheckpointMarkCoder()

getId

public java.lang.String getId()

equals

public boolean equals(java.lang.Object o)

Overrides:

equals in class java.lang.Object

hashCode

public int hashCode()

Overrides:

hashCode in class java.lang.Object

clearCache

public static void clearCache()