org.apache.beam.sdk.io

Class FileSystems

java.lang.Object

org.apache.beam.sdk.io.FileSystems

public class FileSystems
extends java.lang.Object

Clients facing FileSystem utility.

Field Summary

Fields

Modifier and Type	Field and Description
static java.lang.String	DEFAULT_SCHEME

Constructor Summary

Constructors

Constructor and Description
FileSystems()

Method Summary

Modifier and Type	Method and Description
static void	copy(java.util.List<ResourceId> srcResourceIds, java.util.List<ResourceId> destResourceIds, MoveOptions... moveOptions) Copies a List of file-like resources from one location to another.
static java.nio.channels.WritableByteChannel	create(ResourceId resourceId, CreateOptions createOptions) Returns a write channel for the given ResourceId with CreateOptions.
static java.nio.channels.WritableByteChannel	create(ResourceId resourceId, java.lang.String mimeType) Returns a write channel for the given ResourceId.
static void	delete(java.util.Collection<ResourceId> resourceIds, MoveOptions... moveOptions) Deletes a collection of resources.
static boolean	hasGlobWildcard(java.lang.String spec) Checks whether the given spec contains a glob wildcard character.
static java.util.List<MatchResult>	match(java.util.List<java.lang.String> specs) This is the entry point to convert user-provided specs to ResourceIds.
static java.util.List<MatchResult>	match(java.util.List<java.lang.String> specs, EmptyMatchTreatment emptyMatchTreatment) Like match(List), but with a configurable EmptyMatchTreatment.
static MatchResult	match(java.lang.String spec) Like match(List), but for a single resource specification.
static MatchResult	match(java.lang.String spec, EmptyMatchTreatment emptyMatchTreatment) Like match(String), but with a configurable EmptyMatchTreatment.
static ResourceId	matchNewDirectory(java.lang.String singleResourceSpec, java.lang.String... baseNames) Returns a new ResourceId that represents the named directory resource.
static ResourceId	matchNewResource(java.lang.String singleResourceSpec, boolean isDirectory) Returns a new ResourceId that represents the named resource of a type corresponding to the resource type.
static java.util.List<MatchResult>	matchResources(java.util.List<ResourceId> resourceIds) Returns MatchResults for the given resourceIds.
static MatchResult.Metadata	matchSingleFileSpec(java.lang.String spec) Returns the MatchResult.Metadata for a single file resource.
static java.nio.channels.ReadableByteChannel	open(ResourceId resourceId) Returns a read channel for the given ResourceId.
static void	registerFileSystemsOnce(PipelineOptions options) Register file systems once if never done before.
static void	rename(java.util.List<ResourceId> srcResourceIds, java.util.List<ResourceId> destResourceIds, MoveOptions... moveOptions) Renames a List of file-like resources from one location to another.
static void	reportSinkLineage(ResourceId resourceId) Report sink Lineage metrics for resource id.
static void	reportSinkLineage(ResourceId resourceId, FileSystem.LineageLevel level) Report source Lineage metrics for resource id at given level.
static void	reportSourceLineage(ResourceId resourceId) Report source Lineage metrics for resource id.
static void	reportSourceLineage(ResourceId resourceId, FileSystem.LineageLevel level) Report source Lineage metrics for resource id at given level.
static void	setDefaultPipelineOptions(PipelineOptions options) Sets the default configuration in workers.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

DEFAULT_SCHEME

public static final java.lang.String DEFAULT_SCHEME

See Also:

Constant Field Values

Constructor Detail

FileSystems

public FileSystems()

Method Detail

hasGlobWildcard

public static boolean hasGlobWildcard(java.lang.String spec)

Checks whether the given spec contains a glob wildcard character.

match

public static java.util.List<MatchResult> match(java.util.List<java.lang.String> specs)
                                         throws java.io.IOException

This is the entry point to convert user-provided specs to ResourceIds. Callers should use match(java.util.List<java.lang.String>) to resolve users specs ambiguities before calling other methods.

Implementation handles the following ambiguities of a user-provided spec:

1. spec could be a glob or a uri. match(java.util.List<java.lang.String>) should be able to tell and choose efficient implementations.
2. The user-provided spec might refer to files or directories. It is common that users that wish to indicate a directory will omit the trailing path delimiter, such as "/tmp/dir" in Linux. The FileSystem should be able to recognize a directory with the trailing path delimiter omitted, but should always return a correct ResourceId (e.g., "/tmp/dir/" inside the returned MatchResult.

All FileSystem implementations should support glob in the final hierarchical path component of ResourceId. This allows SDK libraries to construct file system agnostic spec. FileSystems can support additional patterns for user-provided specs.

In case the spec schemes don't match any known FileSystem implementations, FileSystems will attempt to use LocalFileSystem to resolve a path.

Specs that do not match any resources are treated according to EmptyMatchTreatment.DISALLOW.

Returns:

List<MatchResult> in the same order of the input specs.

Throws:

java.lang.IllegalArgumentException - if specs are invalid -- empty or have different schemes.

java.io.IOException - if all specs failed to match due to issues like: network connection, authorization. Exception for individual spec is deferred until callers retrieve metadata with MatchResult.metadata().

match

public static java.util.List<MatchResult> match(java.util.List<java.lang.String> specs,
                                                EmptyMatchTreatment emptyMatchTreatment)
                                         throws java.io.IOException

Like match(List), but with a configurable EmptyMatchTreatment.

Throws:

java.io.IOException

match

public static MatchResult match(java.lang.String spec)
                         throws java.io.IOException

Like match(List), but for a single resource specification.

The function match(List) is preferred when matching multiple patterns, as it allows for bulk API calls to remote filesystems.

Throws:

java.io.IOException

match

public static MatchResult match(java.lang.String spec,
                                EmptyMatchTreatment emptyMatchTreatment)
                         throws java.io.IOException

Like match(String), but with a configurable EmptyMatchTreatment.

Throws:

java.io.IOException

matchSingleFileSpec

public static MatchResult.Metadata matchSingleFileSpec(java.lang.String spec)
                                                throws java.io.IOException

Returns the MatchResult.Metadata for a single file resource. Expects a resource specification spec that matches a single result.

Parameters:

spec - a resource specification that matches exactly one result.

Returns:

the MatchResult.Metadata for the specified resource.

Throws:

java.io.FileNotFoundException - if the file resource is not found.

java.io.IOException - in the event of an error in the inner call to match(java.util.List<java.lang.String>), or if the given spec does not match exactly 1 result.

matchResources

public static java.util.List<MatchResult> matchResources(java.util.List<ResourceId> resourceIds)
                                                  throws java.io.IOException

Returns MatchResults for the given resourceIds.

Parameters:

resourceIds - resourceIds that might be derived from match(java.util.List<java.lang.String>), ResourceId.resolve(java.lang.String, org.apache.beam.sdk.io.fs.ResolveOptions), or ResourceId.getCurrentDirectory().

Throws:

java.io.IOException - if all resourceIds failed to match due to issues like: network connection, authorization. Exception for individual ResourceId need to be deferred until callers retrieve metadata with MatchResult.metadata().

create

public static java.nio.channels.WritableByteChannel create(ResourceId resourceId,
                                                           java.lang.String mimeType)
                                                    throws java.io.IOException

Returns a write channel for the given ResourceId.

The resource is not expanded; it is used verbatim.

Parameters:

resourceId - the reference of the file-like resource to create

mimeType - the mine type of the file-like resource to create

Throws:

java.io.IOException

create

public static java.nio.channels.WritableByteChannel create(ResourceId resourceId,
                                                           CreateOptions createOptions)
                                                    throws java.io.IOException

Returns a write channel for the given ResourceId with CreateOptions.

The resource is not expanded; it is used verbatim.

Parameters:

resourceId - the reference of the file-like resource to create

createOptions - the configuration of the create operation

Throws:

java.io.IOException

open

public static java.nio.channels.ReadableByteChannel open(ResourceId resourceId)
                                                  throws java.io.IOException

Returns a read channel for the given ResourceId.

The resource is not expanded; it is used verbatim.

If seeking is supported, then this returns a SeekableByteChannel.

Parameters:

resourceId - the reference of the file-like resource to open

Throws:

java.io.IOException

copy

public static void copy(java.util.List<ResourceId> srcResourceIds,
                        java.util.List<ResourceId> destResourceIds,
                        MoveOptions... moveOptions)
                 throws java.io.IOException

Copies a List of file-like resources from one location to another.

The number of source resources must equal the number of destination resources. Destination resources will be created recursively.

srcResourceIds and destResourceIds must have the same scheme.

It doesn't support copying globs.

Parameters:

srcResourceIds - the references of the source resources

destResourceIds - the references of the destination resources

Throws:

java.io.IOException

rename

public static void rename(java.util.List<ResourceId> srcResourceIds,
                          java.util.List<ResourceId> destResourceIds,
                          MoveOptions... moveOptions)
                   throws java.io.IOException

Renames a List of file-like resources from one location to another.

The number of source resources must equal the number of destination resources. Destination resources will be created recursively.

srcResourceIds and destResourceIds must have the same scheme.

It doesn't support renaming globs.

Src files will be removed, even if the copy is skipped due to specified move options.

Parameters:

srcResourceIds - the references of the source resources

destResourceIds - the references of the destination resources

Throws:

java.io.IOException

delete

public static void delete(java.util.Collection<ResourceId> resourceIds,
                          MoveOptions... moveOptions)
                   throws java.io.IOException

Deletes a collection of resources.

resourceIds must have the same scheme.

Parameters:

resourceIds - the references of the resources to delete.

Throws:

java.io.IOException

reportSourceLineage

public static void reportSourceLineage(ResourceId resourceId)

Report source Lineage metrics for resource id.

reportSinkLineage

public static void reportSinkLineage(ResourceId resourceId)

Report sink Lineage metrics for resource id.

reportSourceLineage

public static void reportSourceLineage(ResourceId resourceId,
                                       FileSystem.LineageLevel level)

Report source Lineage metrics for resource id at given level.

Internal API, no backward compatibility guaranteed.

reportSinkLineage

public static void reportSinkLineage(ResourceId resourceId,
                                     FileSystem.LineageLevel level)

Report source Lineage metrics for resource id at given level.

Internal API, no backward compatibility guaranteed.

setDefaultPipelineOptions

@Internal
public static void setDefaultPipelineOptions(PipelineOptions options)

Sets the default configuration in workers.

It will be used in FileSystemRegistrars for all schemes.

Outside of workers where Beam FileSystem API is used (e.g. test methods, user code executed during pipeline submission), consider use registerFileSystemsOnce(org.apache.beam.sdk.options.PipelineOptions) if initialize FileSystem of supported schema is the main goal.

registerFileSystemsOnce

@Internal
public static void registerFileSystemsOnce(PipelineOptions options)

Register file systems once if never done before.

This method executes setDefaultPipelineOptions(org.apache.beam.sdk.options.PipelineOptions) only if it has never been run, otherwise it returns immediately.

It is internally used by test setup to avoid repeated filesystem registrations (involves expensive ServiceLoader calls) when there are multiple pipeline and PipelineOptions object initialized, which is commonly seen in test execution.

matchNewResource

public static ResourceId matchNewResource(java.lang.String singleResourceSpec,
                                          boolean isDirectory)

Returns a new ResourceId that represents the named resource of a type corresponding to the resource type.

The supplied singleResourceSpec is expected to be in a proper format, including any necessary escaping, for the underlying FileSystem.

This function may throw an IllegalArgumentException if given an invalid argument, such as when the specified singleResourceSpec is not a valid resource name.

matchNewDirectory

public static ResourceId matchNewDirectory(java.lang.String singleResourceSpec,
                                           java.lang.String... baseNames)

Returns a new ResourceId that represents the named directory resource.

Parameters:

singleResourceSpec - the root directory, for example "/abc"

baseNames - a list of named directory, for example ["d", "e", "f"]

Returns:

the ResourceId for the resolved directory. In same example as above, it corresponds to "/abc/d/e/f".