org.apache.beam.sdk.io.thrift

Class ThriftIO

java.lang.Object

org.apache.beam.sdk.io.thrift.ThriftIO

@Experimental(value=SOURCE_SINK)
public class ThriftIO
extends java.lang.Object

PTransforms for reading and writing files containing Thrift encoded data.

Reading Thrift Files

For reading each file in a PCollection of FileIO.ReadableFile, use the readFiles(Class) transform.

For example:

 PCollection<FileIO.ReadableFile> files = pipeline
   .apply(FileIO.match().filepattern(options.getInputFilepattern())
   .apply(FileIO.readMatches());

 PCollection<ExampleType> examples = files.apply(ThriftIO.readFiles(ExampleType.class).withProtocol(thriftProto);
 

Writing Thrift Files

ThriftIO.Sink allows for a PCollection of TBase to be written to Thrift files. It can be used with the general-purpose FileIO transforms with FileIO.write/writeDynamic specifically.

For example:

 pipeline
   .apply(...) // PCollection<ExampleType>
   .apply(FileIO
     .<ExampleType>write()
     .via(ThriftIO.sink(thriftProto))
     .to("destination/path");
 

This IO API is considered experimental and may break or receive backwards-incompatible changes in future versions of the Apache Beam SDK.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	ThriftIO.ReadFiles<T> Implementation of readFiles(java.lang.Class<T>).
static class	ThriftIO.Sink<T extends org.apache.thrift.TBase<?,?>> Implementation of sink(org.apache.thrift.protocol.TProtocolFactory).
protected static class	ThriftIO.ThriftWriter<T extends org.apache.thrift.TBase<?,?>> Writer to write Thrift object to OutputStream.

Method Summary

Modifier and Type	Method and Description
static <T> ThriftIO.ReadFiles<T>	readFiles(java.lang.Class<T> recordClass) Reads each file in a PCollection of FileIO.ReadableFile, which allows more flexible usage.
static <T extends org.apache.thrift.TBase<?,?>> ThriftIO.Sink<T>	sink(org.apache.thrift.protocol.TProtocolFactory factory) Creates a ThriftIO.Sink for use with FileIO.write() and FileIO.writeDynamic().

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

readFiles

public static <T> ThriftIO.ReadFiles<T> readFiles(java.lang.Class<T> recordClass)

Reads each file in a PCollection of FileIO.ReadableFile, which allows more flexible usage.

sink

public static <T extends org.apache.thrift.TBase<?,?>> ThriftIO.Sink<T> sink(org.apache.thrift.protocol.TProtocolFactory factory)

Creates a ThriftIO.Sink for use with FileIO.write() and FileIO.writeDynamic().