TFRecord sources and sinks.
ReadFromTFRecord(file_pattern, coder=BytesCoder, compression_type='auto', validate=True)¶
Transform for reading TFRecord sources.
Initialize a ReadFromTFRecord transform.
- file_pattern – A file glob pattern to read TFRecords from.
- coder – Coder used to decode each record.
- compression_type – Used to handle compressed input files. Default value is CompressionTypes.AUTO, in which case the file_path’s extension will be used to detect the compression.
- validate – Boolean flag to verify that the files exist during the pipeline creation time.
A ReadFromTFRecord transform object.
WriteToTFRecord(file_path_prefix, coder=BytesCoder, file_name_suffix='', num_shards=0, shard_name_template=None, compression_type='auto')¶
Transform for writing to TFRecord sinks.
Initialize WriteToTFRecord transform.
- file_path_prefix – The file path to write to. The files written will begin with this prefix, followed by a shard identifier (see num_shards), and end in a common extension, if given by file_name_suffix.
- coder – Coder used to encode each record.
- file_name_suffix – Suffix for the files written.
- num_shards – The number of files (shards) used for output. If not set, the default value will be used.
- shard_name_template – A template string containing placeholders for the shard number and shard count. When constructing a filename for a particular shard number, the upper-case letters ‘S’ and ‘N’ are replaced with the 0-padded shard number and shard count respectively. This argument can be ‘’ in which case it behaves as if num_shards was set to 1 and only one file will be generated. The default pattern used is ‘-SSSSS-of-NNNNN’ if None is passed as the shard_name_template.
- compression_type – Used to handle compressed output files. Typical value is CompressionTypes.AUTO, in which case the file_path’s extension will be used to detect the compression.
A WriteToTFRecord transform object.