Package org.apache.beam.sdk.io

Class TFRecordIO

java.lang.Object

org.apache.beam.sdk.io.TFRecordIO

public class TFRecordIO
extends Object

PTransforms for reading and writing TensorFlow TFRecord files.

For reading files, use read().

For simple cases of writing files, use write(). For more complex cases (such as ability to write windowed data or writing to multiple destinations) use sink() in combination with FileIO.write() or FileIO.writeDynamic().

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static enum	TFRecordIO.CompressionType	Deprecated. Use Compression.
static class	TFRecordIO.Read	Implementation of read().
static class	TFRecordIO.ReadFiles	Implementation of readFiles().
static class	TFRecordIO.Sink	A FileIO.Sink for use with FileIO.write() and FileIO.writeDynamic().
static class	TFRecordIO.Write	Implementation of write().

Field Summary

Fields

Modifier and Type	Field	Description
static final Coder<byte[]>	DEFAULT_BYTE_ARRAY_CODER	The default coder, which returns each record of the input file as a byte array.

Method Summary

Modifier and Type	Method	Description
static TFRecordIO.Read	read()	A PTransform that reads from a TFRecord file (or multiple TFRecord files matching a pattern) and returns a PCollection containing the decoding of each of the records of the TFRecord file(s) as a byte array.
static TFRecordIO.ReadFiles	readFiles()	Like read(), but reads each file in a PCollection of FileIO.ReadableFile, returned by FileIO.readMatches().
static TFRecordIO.Sink	sink()	Returns a FileIO.Sink for use with FileIO.write() and FileIO.writeDynamic().
static TFRecordIO.Write	write()	A PTransform that writes a PCollection to TFRecord file (or multiple TFRecord files matching a sharding pattern), with each element of the input collection encoded into its own record.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

DEFAULT_BYTE_ARRAY_CODER

public static final Coder<byte[]> DEFAULT_BYTE_ARRAY_CODER

The default coder, which returns each record of the input file as a byte array.

Method Details

read

public static TFRecordIO.Read read()

A PTransform that reads from a TFRecord file (or multiple TFRecord files matching a pattern) and returns a PCollection containing the decoding of each of the records of the TFRecord file(s) as a byte array.

readFiles

public static TFRecordIO.ReadFiles readFiles()

Like read(), but reads each file in a PCollection of FileIO.ReadableFile, returned by FileIO.readMatches().

write

public static TFRecordIO.Write write()

A PTransform that writes a PCollection to TFRecord file (or multiple TFRecord files matching a sharding pattern), with each element of the input collection encoded into its own record.

sink

public static TFRecordIO.Sink sink()

Returns a FileIO.Sink for use with FileIO.write() and FileIO.writeDynamic().