org.apache.beam.sdk.io

Class ReadAllViaFileBasedSourceWithFilename<T>

java.lang.Object

org.apache.beam.sdk.transforms.PTransform<PCollection<FileIO.ReadableFile>,PCollection<T>>

org.apache.beam.sdk.io.ReadAllViaFileBasedSourceTransform<T,KV<java.lang.String,T>>

org.apache.beam.sdk.io.ReadAllViaFileBasedSourceWithFilename<T>

All Implemented Interfaces:

java.io.Serializable, HasDisplayData

public class ReadAllViaFileBasedSourceWithFilename<T>
extends ReadAllViaFileBasedSourceTransform<T,KV<java.lang.String,T>>

Reads each file of the input PCollection and outputs each element as the value of a KV, where the key is the filename from which that value came.

Reads each FileIO.ReadableFile using given parameters for splitting files into offset ranges and for creating a FileBasedSource for a file. The input PCollection must not contain directories.

To obtain the collection of FileIO.ReadableFile from a filepattern, use FileIO.readMatches().

See Also:

Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.beam.sdk.io.ReadAllViaFileBasedSourceTransform

ReadAllViaFileBasedSourceTransform.AbstractReadFileRangesFn<InT,T>, ReadAllViaFileBasedSourceTransform.SplitIntoRangesFn

Field Summary

Fields inherited from class org.apache.beam.sdk.io.ReadAllViaFileBasedSourceTransform

coder, createSource, DEFAULT_USES_RESHUFFLE, desiredBundleSizeBytes, exceptionHandler, usesReshuffle

Fields inherited from class org.apache.beam.sdk.transforms.PTransform

name, resourceHints

Constructor Summary

Constructors

Constructor and Description
ReadAllViaFileBasedSourceWithFilename(long desiredBundleSizeBytes, SerializableFunction<java.lang.String,? extends FileBasedSource<T>> createSource, Coder<KV<java.lang.String,T>> coder)

Method Summary

Modifier and Type	Method and Description
protected DoFn<KV<FileIO.ReadableFile,OffsetRange>,KV<java.lang.String,T>>	readRangesFn()

Methods inherited from class org.apache.beam.sdk.io.ReadAllViaFileBasedSourceTransform

expand

Methods inherited from class org.apache.beam.sdk.transforms.PTransform

compose, compose, getAdditionalInputs, getDefaultOutputCoder, getDefaultOutputCoder, getDefaultOutputCoder, getKindString, getName, getResourceHints, populateDisplayData, setResourceHints, toString, validate, validate

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Detail

ReadAllViaFileBasedSourceWithFilename

public ReadAllViaFileBasedSourceWithFilename(long desiredBundleSizeBytes,
                                             SerializableFunction<java.lang.String,? extends FileBasedSource<T>> createSource,
                                             Coder<KV<java.lang.String,T>> coder)

Method Detail

readRangesFn

protected DoFn<KV<FileIO.ReadableFile,OffsetRange>,KV<java.lang.String,T>> readRangesFn()

Specified by:

readRangesFn in class ReadAllViaFileBasedSourceTransform<T,KV<java.lang.String,T>>