org.apache.beam.sdk.io

Class FileIO

java.lang.Object

org.apache.beam.sdk.io.FileIO

public class FileIO
extends java.lang.Object

Transforms for working with files. Currently includes matching of filepatterns via match() and matchAll(), and reading matches via readMatches().

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	FileIO.Match Implementation of match().
static class	FileIO.MatchAll Implementation of matchAll().
static class	FileIO.MatchConfiguration Describes configuration for matching filepatterns, such as EmptyMatchTreatment and continuous watching for matching files.
static class	FileIO.ReadableFile A utility class for accessing a potentially compressed file.
static class	FileIO.ReadMatches Implementation of readMatches().

Constructor Summary

Constructors

Constructor and Description
FileIO()

Method Summary

Modifier and Type	Method and Description
static FileIO.Match	match() Matches a filepattern using FileSystems.match(java.util.List<java.lang.String>) and produces a collection of matched resources (both files and directories) as MatchResult.Metadata.
static FileIO.MatchAll	matchAll() Like match(), but matches each filepattern in a collection of filepatterns.
static FileIO.ReadMatches	readMatches() Converts each result of match() or matchAll() to a FileIO.ReadableFile which can be used to read the contents of each file, optionally decompressing it.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

FileIO

public FileIO()

Method Detail

match

public static FileIO.Match match()

Matches a filepattern using FileSystems.match(java.util.List<java.lang.String>) and produces a collection of matched resources (both files and directories) as MatchResult.Metadata.

By default, matches the filepattern once and produces a bounded PCollection. To continuously watch the filepattern for new matches, use MatchAll#continuously(Duration, TerminationCondition) - this will produce an unbounded PCollection.

By default, a filepattern matching no resources is treated according to EmptyMatchTreatment.DISALLOW. To configure this behavior, use FileIO.Match.withEmptyMatchTreatment(org.apache.beam.sdk.io.fs.EmptyMatchTreatment).

matchAll

public static FileIO.MatchAll matchAll()

Like match(), but matches each filepattern in a collection of filepatterns.

Resources are not deduplicated between filepatterns, i.e. if the same resource matches multiple filepatterns, it will be produced multiple times.

By default, a filepattern matching no resources is treated according to EmptyMatchTreatment.ALLOW_IF_WILDCARD. To configure this behavior, use FileIO.MatchAll.withEmptyMatchTreatment(org.apache.beam.sdk.io.fs.EmptyMatchTreatment).

readMatches

public static FileIO.ReadMatches readMatches()

Converts each result of match() or matchAll() to a FileIO.ReadableFile which can be used to read the contents of each file, optionally decompressing it.