org.apache.beam.sdk.io

Class ReadAllViaFileBasedSource<T>

java.lang.Object

org.apache.beam.sdk.transforms.PTransform<PCollection<FileIO.ReadableFile>,PCollection<T>>

org.apache.beam.sdk.io.ReadAllViaFileBasedSourceTransform<T,T>

org.apache.beam.sdk.io.ReadAllViaFileBasedSource<T>

All Implemented Interfaces:

java.io.Serializable, HasDisplayData

public class ReadAllViaFileBasedSource<T>
extends ReadAllViaFileBasedSourceTransform<T,T>

Reads each file in the input PCollection of FileIO.ReadableFile using given parameters for splitting files into offset ranges and for creating a FileBasedSource for a file. The input PCollection must not contain directories.

To obtain the collection of FileIO.ReadableFile from a filepattern, use FileIO.readMatches().

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	ReadAllViaFileBasedSource.ReadFileRangesFnExceptionHandler A class to handle errors which occur during file reads.

Nested classes/interfaces inherited from class org.apache.beam.sdk.io.ReadAllViaFileBasedSourceTransform

ReadAllViaFileBasedSourceTransform.AbstractReadFileRangesFn<InT,T>, ReadAllViaFileBasedSourceTransform.SplitIntoRangesFn

Field Summary

Fields inherited from class org.apache.beam.sdk.io.ReadAllViaFileBasedSourceTransform

coder, createSource, DEFAULT_USES_RESHUFFLE, desiredBundleSizeBytes, exceptionHandler, usesReshuffle

Fields inherited from class org.apache.beam.sdk.transforms.PTransform

annotations, displayData, name, resourceHints

Constructor Summary

Constructors

Constructor and Description
ReadAllViaFileBasedSource(long desiredBundleSizeBytes, SerializableFunction<java.lang.String,? extends FileBasedSource<T>> createSource, Coder<T> coder)
ReadAllViaFileBasedSource(long desiredBundleSizeBytes, SerializableFunction<java.lang.String,? extends FileBasedSource<T>> createSource, Coder<T> coder, boolean usesReshuffle, ReadAllViaFileBasedSource.ReadFileRangesFnExceptionHandler exceptionHandler)

Method Summary

Modifier and Type	Method and Description
protected DoFn<KV<FileIO.ReadableFile,OffsetRange>,T>	readRangesFn()

Methods inherited from class org.apache.beam.sdk.io.ReadAllViaFileBasedSourceTransform

expand

Methods inherited from class org.apache.beam.sdk.transforms.PTransform

addAnnotation, compose, compose, getAdditionalInputs, getAnnotations, getDefaultOutputCoder, getDefaultOutputCoder, getDefaultOutputCoder, getKindString, getName, getResourceHints, populateDisplayData, setDisplayData, setResourceHints, toString, validate, validate

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Detail

ReadAllViaFileBasedSource

public ReadAllViaFileBasedSource(long desiredBundleSizeBytes,
                                 SerializableFunction<java.lang.String,? extends FileBasedSource<T>> createSource,
                                 Coder<T> coder)

ReadAllViaFileBasedSource

public ReadAllViaFileBasedSource(long desiredBundleSizeBytes,
                                 SerializableFunction<java.lang.String,? extends FileBasedSource<T>> createSource,
                                 Coder<T> coder,
                                 boolean usesReshuffle,
                                 ReadAllViaFileBasedSource.ReadFileRangesFnExceptionHandler exceptionHandler)

Method Detail

readRangesFn

protected DoFn<KV<FileIO.ReadableFile,OffsetRange>,T> readRangesFn()

Specified by:

readRangesFn in class ReadAllViaFileBasedSourceTransform<T,T>