org.apache.beam.runners.spark.io

Class MicrobatchSource<T,CheckpointMarkT extends UnboundedSource.CheckpointMark>

java.lang.Object

org.apache.beam.sdk.io.Source<T>

org.apache.beam.runners.spark.io.MicrobatchSource<T,CheckpointMarkT>

All Implemented Interfaces:

java.io.Serializable, HasDisplayData

public class MicrobatchSource<T,CheckpointMarkT extends UnboundedSource.CheckpointMark>
extends Source<T>

A Source that accommodates Spark's micro-batch oriented nature and wraps an UnboundedSource.

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
class	MicrobatchSource.Reader Mostly based on BoundedReadFromUnboundedSource's UnboundedToBoundedSourceAdapter, with some adjustments for Spark specifics.

Method Summary

Modifier and Type	Method and Description
static void	clearCache()
boolean	equals(java.lang.Object o)
Coder<CheckpointMarkT>	getCheckpointMarkCoder()
Coder<T>	getDefaultOutputCoder() Returns the default Coder to use for the data read from this source.
java.lang.String	getId()
Source.Reader<T>	getOrCreateReader(PipelineOptions options, CheckpointMarkT checkpointMark)
int	hashCode()
void	validate() Checks that this source is valid, before it can be used in a pipeline.

Methods inherited from class org.apache.beam.sdk.io.Source

populateDisplayData

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, toString, wait, wait, wait

Method Detail

getOrCreateReader

public Source.Reader<T> getOrCreateReader(PipelineOptions options,
                                          CheckpointMarkT checkpointMark)
                                   throws java.io.IOException

Throws:

java.io.IOException

validate

public void validate()

Description copied from class: Source

Checks that this source is valid, before it can be used in a pipeline.

It is recommended to use Preconditions for implementing this method.

Specified by:

validate in class Source<T>

getDefaultOutputCoder

public Coder<T> getDefaultOutputCoder()

Description copied from class: Source

Returns the default Coder to use for the data read from this source.

Specified by:

getDefaultOutputCoder in class Source<T>

getCheckpointMarkCoder

public Coder<CheckpointMarkT> getCheckpointMarkCoder()

getId

public java.lang.String getId()

equals

public boolean equals(java.lang.Object o)

Overrides:

equals in class java.lang.Object

hashCode

public int hashCode()

Overrides:

hashCode in class java.lang.Object

clearCache

public static void clearCache()