K
- Type of keys to be read.V
- Type of values to be read.public static class HadoopFormatIO.HadoopInputFormatBoundedSource<K,V> extends BoundedSource<KV<K,V>> implements java.io.Serializable
HadoopFormatIO
.BoundedSource.BoundedReader<T>
Source.Reader<T>
Modifier | Constructor and Description |
---|---|
protected |
HadoopInputFormatBoundedSource(SerializableConfiguration conf,
Coder<K> keyCoder,
Coder<V> valueCoder,
@Nullable SimpleFunction<?,K> keyTranslationFunction,
@Nullable SimpleFunction<?,V> valueTranslationFunction,
HadoopFormatIO.SerializableSplit inputSplit,
boolean skipKeyClone,
boolean skipValueClone) |
Modifier and Type | Method and Description |
---|---|
protected void |
createInputFormatInstance()
Creates instance of InputFormat class.
|
BoundedSource.BoundedReader<KV<K,V>> |
createReader(PipelineOptions options)
Returns a new
BoundedSource.BoundedReader that reads from this source. |
SerializableConfiguration |
getConfiguration() |
long |
getEstimatedSizeBytes(PipelineOptions po)
An estimate of the total size (in bytes) of the data that would be read from this source.
|
Coder<KV<K,V>> |
getOutputCoder()
Returns the
Coder to use for the data read from this source. |
void |
populateDisplayData(DisplayData.Builder builder)
Register display data for the given transform or component.
|
java.util.List<BoundedSource<KV<K,V>>> |
split(long desiredBundleSizeBytes,
PipelineOptions options)
Splits the source into bundles of approximately
desiredBundleSizeBytes . |
void |
validate()
Checks that this source is valid, before it can be used in a pipeline.
|
getDefaultOutputCoder
protected HadoopInputFormatBoundedSource(SerializableConfiguration conf, Coder<K> keyCoder, Coder<V> valueCoder, @Nullable SimpleFunction<?,K> keyTranslationFunction, @Nullable SimpleFunction<?,V> valueTranslationFunction, HadoopFormatIO.SerializableSplit inputSplit, boolean skipKeyClone, boolean skipValueClone)
public SerializableConfiguration getConfiguration()
public void validate()
Source
It is recommended to use Preconditions
for implementing this method.
public void populateDisplayData(DisplayData.Builder builder)
Source
populateDisplayData(DisplayData.Builder)
is invoked by Pipeline runners to collect
display data via DisplayData.from(HasDisplayData)
. Implementations may call super.populateDisplayData(builder)
in order to register display data in the current namespace,
but should otherwise use subcomponent.populateDisplayData(builder)
to use the namespace
of the subcomponent.
By default, does not register any display data. Implementors may override this method to provide their own display data.
populateDisplayData
in interface HasDisplayData
populateDisplayData
in class Source<KV<K,V>>
builder
- The builder to populate with display data.HasDisplayData
public java.util.List<BoundedSource<KV<K,V>>> split(long desiredBundleSizeBytes, PipelineOptions options) throws java.lang.Exception
BoundedSource
desiredBundleSizeBytes
.split
in class BoundedSource<KV<K,V>>
java.lang.Exception
public long getEstimatedSizeBytes(PipelineOptions po) throws java.lang.Exception
BoundedSource
If there is no way to estimate the size of the source implementations MAY return 0L.
getEstimatedSizeBytes
in class BoundedSource<KV<K,V>>
java.lang.Exception
protected void createInputFormatInstance() throws java.io.IOException
java.io.IOException
public Coder<KV<K,V>> getOutputCoder()
Source
Coder
to use for the data read from this source.getOutputCoder
in class Source<KV<K,V>>
public BoundedSource.BoundedReader<KV<K,V>> createReader(PipelineOptions options) throws java.io.IOException
BoundedSource
BoundedSource.BoundedReader
that reads from this source.createReader
in class BoundedSource<KV<K,V>>
java.io.IOException