org.apache.beam.sdk.io.parquet

Class ParquetIO.Sink

java.lang.Object

org.apache.beam.sdk.io.parquet.ParquetIO.Sink

All Implemented Interfaces:

java.io.Serializable, FileIO.Sink<GenericRecord>

Enclosing class:

ParquetIO

public abstract static class ParquetIO.Sink
extends java.lang.Object
implements FileIO.Sink<GenericRecord>

Implementation of ParquetIO.sink(org.apache.avro.Schema).

See Also:

Serialized Form

Constructor Summary

Constructors

Constructor and Description
Sink()

Method Summary

Modifier and Type	Method and Description
void	flush() Flushes the buffered state (if any) before the channel is closed.
void	open(java.nio.channels.WritableByteChannel channel) Initializes writing to the given channel.
ParquetIO.Sink	withAvroDataModel(GenericData model) Define the Avro data model; see AvroParquetWriter.Builder#withDataModel(GenericData).
ParquetIO.Sink	withCompressionCodec(org.apache.parquet.hadoop.metadata.CompressionCodecName compressionCodecName) Specifies compression codec.
ParquetIO.Sink	withConfiguration(org.apache.hadoop.conf.Configuration configuration) Specify Hadoop configuration for ParquetWriter.
ParquetIO.Sink	withConfiguration(java.util.Map<java.lang.String,java.lang.String> configuration) Specify Hadoop configuration for ParquetWriter.
ParquetIO.Sink	withRowGroupSize(int rowGroupSize) Specify row-group size; if not set or zero, a default is used by the underlying writer.
void	write(GenericRecord element) Appends a single element to the file.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

Sink

public Sink()

Method Detail

withCompressionCodec

public ParquetIO.Sink withCompressionCodec(org.apache.parquet.hadoop.metadata.CompressionCodecName compressionCodecName)

Specifies compression codec. By default, CompressionCodecName.SNAPPY.

withConfiguration

public ParquetIO.Sink withConfiguration(java.util.Map<java.lang.String,java.lang.String> configuration)

Specify Hadoop configuration for ParquetWriter.

withConfiguration

public ParquetIO.Sink withConfiguration(org.apache.hadoop.conf.Configuration configuration)

Specify Hadoop configuration for ParquetWriter.

withRowGroupSize

public ParquetIO.Sink withRowGroupSize(int rowGroupSize)

Specify row-group size; if not set or zero, a default is used by the underlying writer.

withAvroDataModel

public ParquetIO.Sink withAvroDataModel(GenericData model)

Define the Avro data model; see AvroParquetWriter.Builder#withDataModel(GenericData).

open

public void open(java.nio.channels.WritableByteChannel channel)
          throws java.io.IOException

Description copied from interface: FileIO.Sink

Initializes writing to the given channel. Will be invoked once on a given FileIO.Sink instance.

Specified by:

open in interface FileIO.Sink<GenericRecord>

Throws:

java.io.IOException

write

public void write(GenericRecord element)
           throws java.io.IOException

Description copied from interface: FileIO.Sink

Appends a single element to the file. May be invoked zero or more times.

Specified by:

write in interface FileIO.Sink<GenericRecord>

Throws:

java.io.IOException

flush

public void flush()
           throws java.io.IOException

Description copied from interface: FileIO.Sink

Flushes the buffered state (if any) before the channel is closed. Does not need to close the channel. Will be invoked once.

Specified by:

flush in interface FileIO.Sink<GenericRecord>

Throws:

java.io.IOException