org.apache.beam.sdk.io

Interface Summary
Interface	Description
AvroIO.RecordFormatter<ElementT>	Formats an element of a user type into a record with the given schema.
CompressedSource.DecompressingChannelFactory	Factory interface for creating channels that decompress the content of an underlying channel.
FileBasedSink.OutputFileHints	Provides hints about how to generate output files, such as a suggested filename suffix (e.g.
FileBasedSink.WritableByteChannelFactory	Implementations create instances of `WritableByteChannel` used by `FileBasedSink` and related classes to allow decorating, or otherwise transforming, the raw data that would normally be written directly to the `WritableByteChannel` passed into `FileBasedSink.WritableByteChannelFactory.create(WritableByteChannel)`.
FileIO.Sink<ElementT>	Specifies how to write elements to individual files in `FileIO.write()` and `FileIO.writeDynamic()`.
FileSystemRegistrar	A registrar that creates `FileSystem` instances from `PipelineOptions`.
UnboundedSource.CheckpointMark	A marker representing the progress and state of an `UnboundedSource.UnboundedReader`.

Class Summary
Class	Description
AvroIO	`PTransform`s for reading and writing Avro files.
AvroIO.Parse<T>	Implementation of `AvroIO.parseGenericRecords(org.apache.beam.sdk.transforms.SerializableFunction<org.apache.avro.generic.GenericRecord, T>)`.
AvroIO.ParseAll<T>	Implementation of `AvroIO.parseAllGenericRecords(org.apache.beam.sdk.transforms.SerializableFunction<org.apache.avro.generic.GenericRecord, T>)`.
AvroIO.Read<T>	Implementation of `AvroIO.read(java.lang.Class<T>)` and `AvroIO.readGenericRecords(org.apache.avro.Schema)`.
AvroIO.ReadAll<T>	Implementation of `AvroIO.readAll(java.lang.Class<T>)`.
AvroIO.Sink<ElementT>	Implementation of `AvroIO.sink(java.lang.Class<ElementT>)` and `AvroIO.sinkViaGenericRecords(org.apache.avro.Schema, org.apache.beam.sdk.io.AvroIO.RecordFormatter<ElementT>)`.
AvroIO.TypedWrite<UserT,DestinationT,OutputT>	Implementation of `AvroIO.write(java.lang.Class<T>)`.
AvroIO.Write<T>	This class is used as the default return value of `AvroIO.write(java.lang.Class<T>)`
AvroSource<T>	Do not use in pipelines directly: most users should use `AvroIO.Read`.
AvroSource.AvroReader<T>	A `BlockBasedSource.BlockBasedReader` for reading blocks from Avro files.
BlockBasedSource<T>	A `BlockBasedSource` is a `FileBasedSource` where a file consists of blocks of records.
BlockBasedSource.Block<T>	A `Block` represents a block of records that can be read.
BlockBasedSource.BlockBasedReader<T>	A `Reader` that reads records from a `BlockBasedSource`.
BoundedReadFromUnboundedSource<T>	`PTransform` that reads a bounded amount of data from an `UnboundedSource`, specified as one or both of a maximum number of elements or a maximum period of time to read.
BoundedSource<T>	A `Source` that reads a finite amount of input and, because of that, supports some additional operations.
BoundedSource.BoundedReader<T>	A `Reader` that reads a bounded amount of input and supports some additional operations, such as progress estimation and dynamic work rebalancing.
CompressedSource<T>	A Source that reads from compressed files.
CompressedSource.CompressedReader<T>	Reader for a `CompressedSource`.
CountingSource	Most users should use `GenerateSequence` instead.
CountingSource.CounterMark	The checkpoint for an unbounded `CountingSource` is simply the last value produced.
DefaultFilenamePolicy	A default `FileBasedSink.FilenamePolicy` for windowed and unwindowed files.
DefaultFilenamePolicy.Params	Encapsulates constructor parameters to `DefaultFilenamePolicy`.
DefaultFilenamePolicy.ParamsCoder	A Coder for `DefaultFilenamePolicy.Params`.
DynamicAvroDestinations<UserT,DestinationT,OutputT>	A specialization of `FileBasedSink.DynamicDestinations` for `AvroIO`.
DynamicFileDestinations	Some helper classes that derive from `FileBasedSink.DynamicDestinations`.
FileBasedSink<UserT,DestinationT,OutputT>	Abstract class for file-based output.
FileBasedSink.DynamicDestinations<UserT,DestinationT,OutputT>	A class that allows value-dependent writes in `FileBasedSink`.
FileBasedSink.FilenamePolicy	A naming policy for output files.
FileBasedSink.FileResult<DestinationT>	Result of a single bundle write.
FileBasedSink.FileResultCoder<DestinationT>	A coder for `FileBasedSink.FileResult` objects.
FileBasedSink.WriteOperation<DestinationT,OutputT>	Abstract operation that manages the process of writing to `FileBasedSink`.
FileBasedSink.Writer<DestinationT,OutputT>	Abstract writer that writes a bundle to a `FileBasedSink`.
FileBasedSource<T>	A common base class for all file-based `Source`s.
FileBasedSource.FileBasedReader<T>	A `reader` that implements code common to readers of `FileBasedSource`s.
FileIO	General-purpose transforms for working with files: listing files (matching), reading and writing.
FileIO.Match	Implementation of `FileIO.match()`.
FileIO.MatchAll	Implementation of `FileIO.matchAll()`.
FileIO.MatchConfiguration	Describes configuration for matching filepatterns, such as `EmptyMatchTreatment` and continuous watching for matching files.
FileIO.ReadableFile	A utility class for accessing a potentially compressed file.
FileIO.ReadMatches	Implementation of `FileIO.readMatches()`.
FileIO.Write<DestinationT,UserT>	Implementation of `FileIO.write()` and `FileIO.writeDynamic()`.
FileSystem<ResourceIdT extends ResourceId>	File system interface in Beam.
FileSystems	Clients facing `FileSystem` utility.
GenerateSequence	A `PTransform` that produces longs starting from the given value, and either up to the given limit or until `Long.MAX_VALUE` / until the given time elapses.
LocalFileSystemRegistrar	`AutoService` registrar for the `LocalFileSystem`.
LocalResources	Helper functions for producing a `ResourceId` that references a local file or directory.
OffsetBasedSource<T>	A `BoundedSource` that uses offsets to define starting and ending positions.
OffsetBasedSource.OffsetBasedReader<T>	A `Source.Reader` that implements code common to readers of all `OffsetBasedSource`s.
Read	A `PTransform` for reading from a `Source`.
Read.Bounded<T>	`PTransform` that reads from a `BoundedSource`.
Read.Builder	Helper class for building `Read` transforms.
Read.Unbounded<T>	`PTransform` that reads from a `UnboundedSource`.
ReadableFileCoder	A `Coder` for `FileIO.ReadableFile`.
ReadAllViaFileBasedSource<T>	Reads each file in the input `PCollection` of `FileIO.ReadableFile` using given parameters for splitting files into offset ranges and for creating a `FileBasedSource` for a file.
ShardNameTemplate	Standard shard naming templates.
Source<T>	Base class for defining input formats and creating a `Source` for reading the input.
Source.Reader<T>	The interface that readers of custom input sources must implement.
TextIO	`PTransform`s for reading and writing text files.
TextIO.Read	Implementation of `TextIO.read()`.
TextIO.ReadAll	Implementation of `TextIO.readAll()`.
TextIO.ReadFiles	Implementation of `TextIO.readFiles()`.
TextIO.Sink	Implementation of `TextIO.sink()`.
TextIO.TypedWrite<UserT,DestinationT>	Implementation of `TextIO.write()`.
TextIO.Write	This class is used as the default return value of `TextIO.write()`.
TFRecordIO	`PTransform`s for reading and writing TensorFlow TFRecord files.
TFRecordIO.Read	Implementation of `TFRecordIO.read()`.
TFRecordIO.Sink	A `FileIO.Sink` for use with `FileIO.write()` and `FileIO.writeDynamic()`.
TFRecordIO.Write	Implementation of `TFRecordIO.write()`.
UnboundedSource<OutputT,CheckpointMarkT extends UnboundedSource.CheckpointMark>	A `Source` that reads an unbounded amount of input and, because of that, supports some additional operations such as checkpointing, watermarks, and record ids.
UnboundedSource.CheckpointMark.NoopCheckpointMark	A checkpoint mark that does nothing when finalized.
UnboundedSource.UnboundedReader<OutputT>	A `Reader` that reads an unbounded amount of input.
WriteFiles<UserT,DestinationT,OutputT>	A `PTransform` that writes to a `FileBasedSink`.
WriteFilesResult<DestinationT>	The result of a `WriteFiles` transform.

Enum Summary
Enum	Description
CompressedSource.CompressionMode	Deprecated Use `Compression` instead
Compression	Various compression types for reading/writing files.
FileBasedSink.CompressionType	Deprecated use `Compression`.
FileBasedSource.Mode	A given `FileBasedSource` represents a file resource of one of these types.
TextIO.CompressionType	Deprecated Use `Compression`.
TFRecordIO.CompressionType	Deprecated Use `Compression`.

Package org.apache.beam.sdk.io Description

Defines transforms for reading and writing common storage formats, including AvroIO, and TextIO.

The classes in this package provide Read transforms that create PCollections from existing storage:


 PCollection<TableRow> inputData = pipeline.apply(
     BigQueryIO.readTableRows().from("clouddataflow-readonly:samples.weather_stations"));

and Write transforms that persist PCollections to external storage:

 
 PCollection<Integer> numbers = ...;
 numbers.apply(TextIO.write().to("gs://my_bucket/path/to/numbers"));