apache_beam.io.filesystem module

File system abstraction for file-based sources and sinks.

Note to implementors:

“path” arguments will be URLs in the form scheme://foo/bar. The exception is LocalFileSystem, which gets unix-style paths in the form /foo/bar.

class apache_beam.io.filesystem.CompressionTypes

Bases: future.types.newobject.newobject

Enum-like class representing known compression types.

AUTO = 'auto'

BZIP2 = 'bzip2'

DEFLATE = 'deflate'

GZIP = 'gzip'

UNCOMPRESSED = 'uncompressed'

classmethod is_valid_compression_type(compression_type)

Returns True for valid compression types, False otherwise.

classmethod mime_type(compression_type, default='application/octet-stream')

classmethod detect_compression_type(file_path)

Returns the compression type of a file (based on its suffix).

class apache_beam.io.filesystem.CompressedFile(fileobj, compression_type='gzip', read_size=16777216)

Bases: future.types.newobject.newobject

File wrapper for easier handling of compressed files.

readable()

writeable()

write(data)

Write data to file.

read(num_bytes)

readline()

Equivalent to standard file.readline(). Same return conventions apply.

closed()

close()

flush()

seekable

seek(offset, whence=0)

Set the file’s current offset.

Seeking behavior:

• seeking from the end os.SEEK_END the whole file is decompressed once to determine its size. Therefore it is preferred to use os.SEEK_SET or os.SEEK_CUR to avoid the processing overhead
• seeking backwards from the current position rewinds the file to 0 and decompresses the chunks to the requested offset
• seeking is only supported in files opened for reading
• if the new offset is out of bound, it is adjusted to either 0 or EOF.

Parameters:

• offset (int) – seek offset in the uncompressed content represented as number
• whence (int) – seek mode. Supported modes are os.SEEK_SET (absolute seek), os.SEEK_CUR (seek relative to the current position), and os.SEEK_END (seek relative to the end, offset should be negative).

Raises:

• IOError – When this buffer is closed.
• ValueError – When whence is invalid or the file is not seekable

tell()

Returns current position in uncompressed file.

class apache_beam.io.filesystem.FileMetadata(path, size_in_bytes)

Bases: future.types.newobject.newobject

Metadata about a file path that is the output of FileSystem.match

class apache_beam.io.filesystem.MatchResult(pattern, metadata_list)

Bases: future.types.newobject.newobject

Result from the FileSystem match operation which contains the list of matched FileMetadata.

class apache_beam.io.filesystem.FileSystem(pipeline_options)

Bases: apache_beam.utils.plugin.BeamPlugin

A class that defines the functions that can be performed on a filesystem.

All methods are abstract and they are for file system providers to implement. Clients should use the FileSystems class to interact with the correct file system based on the provided file pattern scheme.

Parameters:pipeline_options – Instance of PipelineOptions or dict of options and values (like RuntimeValueProvider.runtime_options).

CHUNK_SIZE = 1

classmethod scheme()

URI scheme for the FileSystem

join(basepath, *paths)

Join two or more pathname components for the filesystem

Parameters:

• basepath – string path of the first component of the path
• paths – path components to be added

Returns: full path after combining all the passed components

split(path)

Splits the given path into two parts.

Splits the path into a pair (head, tail) such that tail contains the last component of the path and head contains everything up to that.

For file-systems other than the local file-system, head should include the prefix.

Parameters:path – path as a string Returns:a pair of path components as strings.

mkdirs(path)

Recursively create directories for the provided path.

Parameters:path – string path of the directory structure that should be created Raises:IOError if leaf directory already exists.

has_dirs()

Whether this FileSystem supports directories.

match_files(file_metas, pattern)

Filter FileMetadata objects by pattern

Parameters:

• file_metas (list of FileMetadata) – Files to consider when matching
• pattern (str) – File pattern

See also

translate_pattern()

Returns:Generator of matching FileMetadata

static translate_pattern(pattern)

Translate a pattern to a regular expression. There is no way to quote meta-characters.

Pattern syntax:

The pattern syntax is based on the fnmatch syntax, with the following differences:

• * Is equivalent to [^/\]* rather than .*.
• ** Is equivalent to .*.

See also

match() uses this method

This method is based on Python 2.7’s fnmatch.translate. The code in this method is licensed under PYTHON SOFTWARE FOUNDATION LICENSE VERSION 2.

match(patterns, limits=None)

Find all matching paths to the patterns provided.

See also

translate_pattern()

Patterns ending with ‘/’ will be appended with ‘*’.

Parameters:

• patterns – list of string for the file path pattern to match against
• limits – list of maximum number of responses that need to be fetched

Returns: list of MatchResult objects.

Raises:BeamIOError if any of the pattern match operations fail

create(path, mime_type='application/octet-stream', compression_type='auto')

Returns a write channel for the given file path.

Parameters:

• path – string path of the file object to be written to the system
• mime_type – MIME type to specify the type of content in the file object
• compression_type – Type of compression to be used for this object

Returns: file handle with a close function for the user to use

open(path, mime_type='application/octet-stream', compression_type='auto')

Returns a read channel for the given file path.

Parameters:

• path – string path of the file object to be read
• mime_type – MIME type to specify the type of content in the file object
• compression_type – Type of compression to be used for this object

Returns: file handle with a close function for the user to use

copy(source_file_names, destination_file_names)

Recursively copy the file tree from the source to the destination

Parameters:

• source_file_names – list of source file objects that needs to be copied
• destination_file_names – list of destination of the new object

Raises:

BeamIOError if any of the copy operations fail

rename(source_file_names, destination_file_names)

Rename the files at the source list to the destination list. Source and destination lists should be of the same size.

Parameters:

• source_file_names – List of file paths that need to be moved
• destination_file_names – List of destination_file_names for the files

Raises:

BeamIOError if any of the rename operations fail

exists(path)

Check if the provided path exists on the FileSystem.

Parameters:path – string path that needs to be checked.

Returns: boolean flag indicating if path exists

size(path)

Get size in bytes of a file on the FileSystem.

Parameters:path – string filepath of file.

Returns: int size of file according to the FileSystem.

Raises:BeamIOError if path doesn’t exist.

last_updated(path)

Get UNIX Epoch time in seconds on the FileSystem.

Parameters:path – string path of file.

Returns: float UNIX Epoch time

Raises:BeamIOError if path doesn’t exist.

checksum(path)

Fetch checksum metadata of a file on the FileSystem.

This operation returns checksum metadata as stored in the underlying FileSystem. It should not need to read file data to obtain this value. Checksum type and format are FileSystem dependent and are not compatible between FileSystems. FileSystem implementations may return file size if a checksum isn’t available.

Parameters:path – string path of a file.

Returns: string containing checksum

Raises:BeamIOError if path isn’t a file or doesn’t exist.

delete(paths)

Deletes files or directories at the provided paths. Directories will be deleted recursively.

Parameters:paths – list of paths that give the file objects to be deleted Raises:BeamIOError if any of the delete operations fail