public class ReadAllViaFileBasedSourceWithFilename<T> extends ReadAllViaFileBasedSourceTransform<T,KV<java.lang.String,T>>
PCollection
and outputs each element as the value of a
KV
, where the key is the filename from which that value came.
Reads each FileIO.ReadableFile
using given parameters for splitting files into offset
ranges and for creating a FileBasedSource
for a file. The input PCollection
must
not contain directories
.
To obtain the collection of FileIO.ReadableFile
from a filepattern, use FileIO.readMatches()
.
ReadAllViaFileBasedSourceTransform.AbstractReadFileRangesFn<InT,T>, ReadAllViaFileBasedSourceTransform.SplitIntoRangesFn
coder, createSource, DEFAULT_USES_RESHUFFLE, desiredBundleSizeBytes, exceptionHandler, usesReshuffle
annotations, displayData, name, resourceHints
Constructor and Description |
---|
ReadAllViaFileBasedSourceWithFilename(long desiredBundleSizeBytes,
SerializableFunction<java.lang.String,? extends FileBasedSource<T>> createSource,
Coder<KV<java.lang.String,T>> coder) |
Modifier and Type | Method and Description |
---|---|
protected DoFn<KV<FileIO.ReadableFile,OffsetRange>,KV<java.lang.String,T>> |
readRangesFn() |
expand
addAnnotation, compose, compose, getAdditionalInputs, getAnnotations, getDefaultOutputCoder, getDefaultOutputCoder, getDefaultOutputCoder, getKindString, getName, getResourceHints, populateDisplayData, setDisplayData, setResourceHints, toString, validate, validate
public ReadAllViaFileBasedSourceWithFilename(long desiredBundleSizeBytes, SerializableFunction<java.lang.String,? extends FileBasedSource<T>> createSource, Coder<KV<java.lang.String,T>> coder)
protected DoFn<KV<FileIO.ReadableFile,OffsetRange>,KV<java.lang.String,T>> readRangesFn()
readRangesFn
in class ReadAllViaFileBasedSourceTransform<T,KV<java.lang.String,T>>