Package org.apache.beam.sdk.io.thrift

Class ThriftIO

java.lang.Object

org.apache.beam.sdk.io.thrift.ThriftIO

public class ThriftIO
extends Object

PTransforms for reading and writing files containing Thrift encoded data.

Reading Thrift Files

For reading each file in a PCollection of FileIO.ReadableFile, use the readFiles(Class) transform.

For example:

 PCollection<FileIO.ReadableFile> files = pipeline
   .apply(FileIO.match().filepattern(options.getInputFilepattern())
   .apply(FileIO.readMatches());

 PCollection<ExampleType> examples = files.apply(ThriftIO.readFiles(ExampleType.class).withProtocol(thriftProto);
 

Writing Thrift Files

ThriftIO.Sink allows for a PCollection of TBase to be written to Thrift files. It can be used with the general-purpose FileIO transforms with FileIO.write/writeDynamic specifically.

For example:

 pipeline
   .apply(...) // PCollection<ExampleType>
   .apply(FileIO
     .<ExampleType>write()
     .via(ThriftIO.sink(thriftProto))
     .to("destination/path");
 

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static class	ThriftIO.ReadFiles<T>	Implementation of readFiles(java.lang.Class<T>).
static class	ThriftIO.Sink<T extends org.apache.thrift.TBase<?,?>>	Implementation of sink(org.apache.thrift.protocol.TProtocolFactory).
protected static class	ThriftIO.ThriftWriter<T extends org.apache.thrift.TBase<?,?>>	Writer to write Thrift object to OutputStream.

Method Summary

Modifier and Type	Method	Description
static <T> ThriftIO.ReadFiles<T>	readFiles(Class<T> recordClass)	Reads each file in a PCollection of FileIO.ReadableFile, which allows more flexible usage.
static <T extends org.apache.thrift.TBase<?, ?>> ThriftIO.Sink<T>	sink(org.apache.thrift.protocol.TProtocolFactory factory)	Creates a ThriftIO.Sink for use with FileIO.write() and FileIO.writeDynamic().

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Details

readFiles

public static <T> ThriftIO.ReadFiles<T> readFiles(Class<T> recordClass)

Reads each file in a PCollection of FileIO.ReadableFile, which allows more flexible usage.

sink

public static <T extends org.apache.thrift.TBase<?,
?>>
ThriftIO.Sink<T> sink(org.apache.thrift.protocol.TProtocolFactory factory)

Creates a ThriftIO.Sink for use with FileIO.write() and FileIO.writeDynamic().