apache_beam.io.filesystem module

File system abstraction for file-based sources and sinks.

Note to implementors:: “path” arguments will be URLs in the form scheme://foo/bar. The exception is LocalFileSystem, which gets unix-style paths in the form /foo/bar.

class apache_beam.io.filesystem.CompressionTypes[source]

Bases: object

Enum-like class representing known compression types.

AUTO = 'auto'

BZIP2 = 'bzip2'

DEFLATE = 'deflate'

ZSTD = 'zstd'

GZIP = 'gzip'

LZMA = 'lzma'

UNCOMPRESSED = 'uncompressed'

classmethod is_valid_compression_type(compression_type)[source]: Returns True for valid compression types, False otherwise.

classmethod mime_type(compression_type, default='application/octet-stream')[source]

classmethod detect_compression_type(file_path)[source]: Returns the compression type of a file (based on its suffix).

class apache_beam.io.filesystem.CompressedFile(fileobj: BinaryIO, compression_type='gzip', read_size=16777216)[source]

Bases: object

File wrapper for easier handling of compressed files.

readable() → bool[source]

writeable() → bool[source]

write(data: bytes) → None[source]: Write data to file.

read(num_bytes: int | None = None) → bytes[source]

readline() → bytes[source]: Equivalent to standard file.readline(). Same return conventions apply.

closed() → bool[source]

close() → None[source]

flush() → None[source]

property seekable: bool

seek(offset: int, whence: int = 0) → None[source]

Set the file’s current offset.

Seeking behavior:

seeking from the end os.SEEK_END the whole file is decompressed once to determine its size. Therefore it is preferred to use os.SEEK_SET or os.SEEK_CUR to avoid the processing overhead

seeking backwards from the current position rewinds the file to 0 and decompresses the chunks to the requested offset

seeking is only supported in files opened for reading

if the new offset is out of bound, it is adjusted to either 0 or EOF.

Parameters:

offset (int) – seek offset in the uncompressed content represented as number
whence (int) – seek mode. Supported modes are os.SEEK_SET (absolute seek), os.SEEK_CUR (seek relative to the current position), and os.SEEK_END (seek relative to the end, offset should be negative).

Raises:

IOError – When this buffer is closed.
ValueError – When whence is invalid or the file is not seekable

tell() → int[source]: Returns current position in uncompressed file.

class apache_beam.io.filesystem.FileMetadata(path: str, size_in_bytes: int, last_updated_in_seconds: float = 0.0)[source]

Bases: object

Metadata about a file path that is the output of FileSystem.match.

Fields:: path: [Required] file path. size_in_bytes: [Required] file size in bytes. last_updated_in_seconds: [Optional] last modified timestamp of the file, or valued 0.0 if not specified.

class apache_beam.io.filesystem.FileSystem(pipeline_options)[source]

Bases: BeamPlugin

A class that defines the functions that can be performed on a filesystem.

All methods are abstract and they are for file system providers to implement. Clients should use the FileSystems class to interact with the correct file system based on the provided file pattern scheme.

Parameters:: pipeline_options – Instance of PipelineOptions or dict of options and values (like RuntimeValueProvider.runtime_options).

CHUNK_SIZE = 1

classmethod scheme()[source]: URI scheme for the FileSystem

abstract join(basepath: str, *paths: str) → str[source]

Join two or more pathname components for the filesystem

Parameters:

basepath – string path of the first component of the path
paths – path components to be added

Returns: full path after combining all the passed components

abstract split(path: str) → Tuple[str, str][source]

Splits the given path into two parts.

Splits the path into a pair (head, tail) such that tail contains the last component of the path and head contains everything up to that.

For file-systems other than the local file-system, head should include the prefix.

Parameters:: path – path as a string
Returns:: a pair of path components as strings.

abstract mkdirs(path)[source]

Recursively create directories for the provided path.

Parameters:: path – string path of the directory structure that should be created
Raises:: IOError – if leaf directory already exists.

abstract has_dirs()[source]: Whether this FileSystem supports directories.

match_files(file_metas: List[FileMetadata], pattern: str) → Iterator[FileMetadata][source]

Filter FileMetadata objects by pattern

Parameters:

file_metas (list of FileMetadata) – Files to consider when matching
pattern (str) – File pattern