Package org.apache.beam.sdk.io.hadoop.format

Class HadoopFormatIO.HadoopInputFormatBoundedSource<K,V>

java.lang.Object

org.apache.beam.sdk.io.Source<KV<K,V>>

org.apache.beam.sdk.io.BoundedSource<KV<K,V>>

org.apache.beam.sdk.io.hadoop.format.HadoopFormatIO.HadoopInputFormatBoundedSource<K,V>

Type Parameters:

K - Type of keys to be read.

V - Type of values to be read.

All Implemented Interfaces:

Serializable, HasDisplayData

Enclosing class:

HadoopFormatIO

public static class HadoopFormatIO.HadoopInputFormatBoundedSource<K,V>
extends BoundedSource<KV<K,V>>
implements Serializable

Bounded source implementation for HadoopFormatIO.

See Also:

• Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.beam.sdk.io.BoundedSource

BoundedSource.BoundedReader<T>

Nested classes/interfaces inherited from class org.apache.beam.sdk.io.Source

Source.Reader<T>

Constructor Summary

Constructors

Modifier	Constructor	Description
protected	HadoopInputFormatBoundedSource(SerializableConfiguration conf, Coder<K> keyCoder, Coder<V> valueCoder, @Nullable SimpleFunction<?,K> keyTranslationFunction, @Nullable SimpleFunction<?,V> valueTranslationFunction, HadoopFormatIO.SerializableSplit inputSplit, boolean skipKeyClone, boolean skipValueClone)

Method Summary

Modifier and Type	Method	Description
protected void	createInputFormatInstance()	Creates instance of InputFormat class.
BoundedSource.BoundedReader<KV<K,V>>	createReader(PipelineOptions options)	Returns a new BoundedSource.BoundedReader that reads from this source.
SerializableConfiguration	getConfiguration()
long	getEstimatedSizeBytes(PipelineOptions po)	An estimate of the total size (in bytes) of the data that would be read from this source.
Coder<KV<K,V>>	getOutputCoder()	Returns the Coder to use for the data read from this source.
void	populateDisplayData(DisplayData.Builder builder)	Register display data for the given transform or component.
List<BoundedSource<KV<K,V>>>	split(long desiredBundleSizeBytes, PipelineOptions options)	Splits the source into bundles of approximately desiredBundleSizeBytes.
void	validate()	Checks that this source is valid, before it can be used in a pipeline.

Methods inherited from class org.apache.beam.sdk.io.Source

getDefaultOutputCoder

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

HadoopInputFormatBoundedSource

protected HadoopInputFormatBoundedSource(SerializableConfiguration conf,
 Coder<K> keyCoder,
 Coder<V> valueCoder,
 @Nullable SimpleFunction<?,K> keyTranslationFunction,
 @Nullable SimpleFunction<?,V> valueTranslationFunction,
 HadoopFormatIO.SerializableSplit inputSplit,
 boolean skipKeyClone,
 boolean skipValueClone)

Method Details

getConfiguration

public SerializableConfiguration getConfiguration()

validate

public void validate()

Description copied from class: Source

Checks that this source is valid, before it can be used in a pipeline.

It is recommended to use Preconditions for implementing this method.

Overrides:

validate in class Source<KV<K,V>>

populateDisplayData

public void populateDisplayData(DisplayData.Builder builder)

Description copied from class: Source

Register display data for the given transform or component.

populateDisplayData(DisplayData.Builder) is invoked by Pipeline runners to collect display data via DisplayData.from(HasDisplayData). Implementations may call super.populateDisplayData(builder) in order to register display data in the current namespace, but should otherwise use subcomponent.populateDisplayData(builder) to use the namespace of the subcomponent.

By default, does not register any display data. Implementors may override this method to provide their own display data.

Specified by:

populateDisplayData in interface HasDisplayData

Overrides:

populateDisplayData in class Source<KV<K,V>>

Parameters:

builder - The builder to populate with display data.

See Also:

• HasDisplayData

split

public List<BoundedSource<KV<K,V>>> split(long desiredBundleSizeBytes,
 PipelineOptions options)
                                   throws Exception

Description copied from class: BoundedSource

Splits the source into bundles of approximately desiredBundleSizeBytes.

Specified by:

split in class BoundedSource<KV<K,V>>

Throws:

Exception

getEstimatedSizeBytes

public long getEstimatedSizeBytes(PipelineOptions po)
                           throws Exception

Description copied from class: BoundedSource

An estimate of the total size (in bytes) of the data that would be read from this source. This estimate is in terms of external storage size, before any decompression or other processing done by the reader.

If there is no way to estimate the size of the source implementations MAY return 0L.

Specified by:

getEstimatedSizeBytes in class BoundedSource<KV<K,V>>

Throws:

Exception

createInputFormatInstance

protected void createInputFormatInstance()
                                  throws IOException

Creates instance of InputFormat class. The InputFormat class name is specified in the Hadoop configuration.

Throws:

IOException

getOutputCoder

public Coder<KV<K,V>> getOutputCoder()

Description copied from class: Source

Returns the Coder to use for the data read from this source.

Overrides:

getOutputCoder in class Source<KV<K,V>>

createReader

public BoundedSource.BoundedReader<KV<K,V>> createReader(PipelineOptions options)
                                                  throws IOException

Description copied from class: BoundedSource

Returns a new BoundedSource.BoundedReader that reads from this source.

Specified by:

createReader in class BoundedSource<KV<K,V>>

Throws:

IOException