org.apache.beam.sdk.io

Class TFRecordIO

java.lang.Object

org.apache.beam.sdk.io.TFRecordIO

public class TFRecordIO
extends java.lang.Object

PTransforms for reading and writing TensorFlow TFRecord files.

For reading files, use read().

For simple cases of writing files, use write(). For more complex cases (such as ability to write windowed data or writing to multiple destinations) use sink() in combination with FileIO.write() or FileIO.writeDynamic().

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	TFRecordIO.CompressionType Deprecated. Use Compression.
static class	TFRecordIO.Read Implementation of read().
static class	TFRecordIO.ReadFiles Implementation of readFiles().
static class	TFRecordIO.Sink A FileIO.Sink for use with FileIO.write() and FileIO.writeDynamic().
static class	TFRecordIO.Write Implementation of write().

Field Summary

Fields

Modifier and Type	Field and Description
static Coder<byte[]>	DEFAULT_BYTE_ARRAY_CODER The default coder, which returns each record of the input file as a byte array.

Method Summary

Modifier and Type	Method and Description
static TFRecordIO.Read	read() A PTransform that reads from a TFRecord file (or multiple TFRecord files matching a pattern) and returns a PCollection containing the decoding of each of the records of the TFRecord file(s) as a byte array.
static TFRecordIO.ReadFiles	readFiles() Like read(), but reads each file in a PCollection of FileIO.ReadableFile, returned by FileIO.readMatches().
static TFRecordIO.Sink	sink() Returns a FileIO.Sink for use with FileIO.write() and FileIO.writeDynamic().
static TFRecordIO.Write	write() A PTransform that writes a PCollection to TFRecord file (or multiple TFRecord files matching a sharding pattern), with each element of the input collection encoded into its own record.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

DEFAULT_BYTE_ARRAY_CODER

public static final Coder<byte[]> DEFAULT_BYTE_ARRAY_CODER

The default coder, which returns each record of the input file as a byte array.

Method Detail

read

public static TFRecordIO.Read read()

A PTransform that reads from a TFRecord file (or multiple TFRecord files matching a pattern) and returns a PCollection containing the decoding of each of the records of the TFRecord file(s) as a byte array.

readFiles

public static TFRecordIO.ReadFiles readFiles()

Like read(), but reads each file in a PCollection of FileIO.ReadableFile, returned by FileIO.readMatches().

write

public static TFRecordIO.Write write()

A PTransform that writes a PCollection to TFRecord file (or multiple TFRecord files matching a sharding pattern), with each element of the input collection encoded into its own record.

sink

public static TFRecordIO.Sink sink()

Returns a FileIO.Sink for use with FileIO.write() and FileIO.writeDynamic().