FileIO.Write (Apache Beam 2.6.0)

java.lang.Object
- org.apache.beam.sdk.transforms.PTransform<PCollection<UserT>,WriteFilesResult<DestinationT>>
- - org.apache.beam.sdk.io.FileIO.Write<DestinationT,UserT>

All Implemented Interfaces:: java.io.Serializable, HasDisplayData

Enclosing class:: FileIO

@Experimental(value=SOURCE_SINK)
public abstract static class FileIO.Write<DestinationT,UserT>
extends PTransform<PCollection<UserT>,WriteFilesResult<DestinationT>>

Implementation of FileIO.write() and FileIO.writeDynamic().

See Also:: Serialized Form

Nested Class Summary

Nested Classes
Modifier and Type Class and Description

static interface FileIO.Write.FileNaming
A policy for generating names for shard files.

Nested Classes
Modifier and Type	Class and Description
`static interface`	`FileIO.Write.FileNaming` A policy for generating names for shard files.

Field Summary
- Fields inherited from class org.apache.beam.sdk.transforms.PTransform
  name

Constructor Summary

Constructors
Constructor and Description

Write()

Constructors
Constructor and Description
`Write()`

Method Summary

All Methods Static Methods Instance Methods Concrete Methods Deprecated Methods
Modifier and Type	Method and Description
`FileIO.Write<DestinationT,UserT>`	`by(Contextful<Contextful.Fn<UserT,DestinationT>> destinationFn)` Like `by(org.apache.beam.sdk.transforms.SerializableFunction<UserT, DestinationT>)`, but with access to context such as side inputs.
`FileIO.Write<DestinationT,UserT>`	`by(SerializableFunction<UserT,DestinationT> destinationFn)` Specifies how to partition elements into groups ("destinations").
`static FileIO.Write.FileNaming`	`defaultNaming(java.lang.String prefix, java.lang.String suffix)`
`static FileIO.Write.FileNaming`	`defaultNaming(ValueProvider<java.lang.String> prefix, ValueProvider<java.lang.String> suffix)`
`WriteFilesResult<DestinationT>`	`expand(PCollection<UserT> input)` Override this method to specify how this `PTransform` should be expanded on the given `InputT`.
`static FileIO.Write.FileNaming`	`relativeFileNaming(ValueProvider<java.lang.String> baseDirectory, FileIO.Write.FileNaming innerNaming)`
`FileIO.Write<DestinationT,UserT>`	`to(java.lang.String directory)` Specifies a common directory for all generated files.
`FileIO.Write<DestinationT,UserT>`	`to(ValueProvider<java.lang.String> directory)` Like `to(String)` but with a `ValueProvider`.
`FileIO.Write<DestinationT,UserT>`	`via(Contextful<Contextful.Fn<DestinationT,FileIO.Sink<UserT>>> sinkFn)` Like `via(Contextful, Contextful)`, but the output type of the sink is the same as the type of the input collection.
`<OutputT> FileIO.Write<DestinationT,UserT>`	`via(Contextful<Contextful.Fn<UserT,OutputT>> outputFn, Contextful<Contextful.Fn<DestinationT,FileIO.Sink<OutputT>>> sinkFn)` Specifies how to create a `FileIO.Sink` for a particular destination and how to map the element type to the sink's output type.
`<OutputT> FileIO.Write<DestinationT,UserT>`	`via(Contextful<Contextful.Fn<UserT,OutputT>> outputFn, FileIO.Sink<OutputT> sink)` Like `via(Contextful, Contextful)`, but uses the same sink for all destinations.
`FileIO.Write<DestinationT,UserT>`	`via(FileIO.Sink<UserT> sink)` Like `via(Contextful)`, but uses the same `FileIO.Sink` for all destinations.
`FileIO.Write<DestinationT,UserT>`	`withCompression(Compression compression)` Specifies to compress all generated shard files using the given `Compression` and, by default, append the respective extension to the filename.
`FileIO.Write<DestinationT,UserT>`	`withDestinationCoder(Coder<DestinationT> destinationCoder)` Specifies a `Coder` for the destination type, if it can not be inferred from `by(org.apache.beam.sdk.transforms.SerializableFunction<UserT, DestinationT>)`.
`FileIO.Write<DestinationT,UserT>`	`withEmptyGlobalWindowDestination(DestinationT emptyWindowDestination)` If `withIgnoreWindowing()` is specified, specifies a destination to be used in case the collection is empty, to generate the (only, empty) output file.
`FileIO.Write<DestinationT,UserT>`	`withIgnoreWindowing()` Deprecated. Avoid usage of this method: its effects are complex and it will be removed in future versions of Beam. Right now it exists for compatibility with `WriteFiles`.
`FileIO.Write<DestinationT,UserT>`	`withNaming(Contextful<Contextful.Fn<DestinationT,FileIO.Write.FileNaming>> namingFn)` Like `withNaming(SerializableFunction)` but allows accessing context, such as side inputs, from the function.
`FileIO.Write<DestinationT,UserT>`	`withNaming(FileIO.Write.FileNaming naming)` Specifies a custom strategy for generating filenames.
`FileIO.Write<DestinationT,UserT>`	`withNaming(SerializableFunction<DestinationT,FileIO.Write.FileNaming> namingFn)` Specifies a custom strategy for generating filenames depending on the destination, similar to `withNaming(FileNaming)`.
`FileIO.Write<DestinationT,UserT>`	`withNumShards(int numShards)` Specifies to use a given fixed number of shards per window.
`FileIO.Write<DestinationT,UserT>`	`withNumShards(ValueProvider<java.lang.Integer> numShards)` Like `withNumShards(int)`.
`FileIO.Write<DestinationT,UserT>`	`withPrefix(java.lang.String prefix)` Specifies a common prefix to use for all generated filenames, if using the default file naming.
`FileIO.Write<DestinationT,UserT>`	`withPrefix(ValueProvider<java.lang.String> prefix)` Like `withPrefix(String)` but with a `ValueProvider`.
`FileIO.Write<DestinationT,UserT>`	`withSharding(PTransform<PCollection<UserT>,PCollectionView<java.lang.Integer>> sharding)` Specifies a `PTransform` to use for computing the desired number of shards in each window.
`FileIO.Write<DestinationT,UserT>`	`withSuffix(java.lang.String suffix)` Specifies a common suffix to use for all generated filenames, if using the default file naming.
`FileIO.Write<DestinationT,UserT>`	`withSuffix(ValueProvider<java.lang.String> suffix)` Like `withSuffix(String)` but with a `ValueProvider`.
`FileIO.Write<DestinationT,UserT>`	`withTempDirectory(java.lang.String tempDirectory)` Specifies a directory into which all temporary files will be placed.
`FileIO.Write<DestinationT,UserT>`	`withTempDirectory(ValueProvider<java.lang.String> tempDirectory)` Like `withTempDirectory(String)`.

Methods inherited from class org.apache.beam.sdk.transforms.PTransform
getAdditionalInputs, getDefaultOutputCoder, getDefaultOutputCoder, getDefaultOutputCoder, getKindString, getName, populateDisplayData, toString, validate

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

- Constructor Detail
  - Write
```
public Write()
```
- Method Detail
  - defaultNaming
```
public static FileIO.Write.FileNaming defaultNaming(java.lang.String prefix,
                                                    java.lang.String suffix)
```
  - defaultNaming
```
public static FileIO.Write.FileNaming defaultNaming(ValueProvider<java.lang.String> prefix,
                                                    ValueProvider<java.lang.String> suffix)
```
  - relativeFileNaming
```
public static FileIO.Write.FileNaming relativeFileNaming(ValueProvider<java.lang.String> baseDirectory,
                                                         FileIO.Write.FileNaming innerNaming)
```
  - by
```
public FileIO.Write<DestinationT,UserT> by(SerializableFunction<UserT,DestinationT> destinationFn)
```
    Specifies how to partition elements into groups ("destinations").
  - by
```
public FileIO.Write<DestinationT,UserT> by(Contextful<Contextful.Fn<UserT,DestinationT>> destinationFn)
```
    Like by(org.apache.beam.sdk.transforms.SerializableFunction<UserT, DestinationT>), but with access to context such as side inputs.
  - via
```
public <OutputT> FileIO.Write<DestinationT,UserT> via(Contextful<Contextful.Fn<UserT,OutputT>> outputFn,
                                                      Contextful<Contextful.Fn<DestinationT,FileIO.Sink<OutputT>>> sinkFn)
```
    Specifies how to create a FileIO.Sink for a particular destination and how to map the element type to the sink's output type. The sink function must create a new FileIO.Sink instance every time it is called.
  - via
```
public <OutputT> FileIO.Write<DestinationT,UserT> via(Contextful<Contextful.Fn<UserT,OutputT>> outputFn,
                                                      FileIO.Sink<OutputT> sink)
```
    Like via(Contextful, Contextful), but uses the same sink for all destinations.
  - via
```
public FileIO.Write<DestinationT,UserT> via(Contextful<Contextful.Fn<DestinationT,FileIO.Sink<UserT>>> sinkFn)
```
    Like via(Contextful, Contextful), but the output type of the sink is the same as the type of the input collection. The sink function must create a new FileIO.Sink instance every time it is called.
  - via
```
public FileIO.Write<DestinationT,UserT> via(FileIO.Sink<UserT> sink)
```
    Like via(Contextful), but uses the same FileIO.Sink for all destinations.
  - to
```
public FileIO.Write<DestinationT,UserT> to(java.lang.String directory)
```
    Specifies a common directory for all generated files. A temporary generated sub-directory of this directory will be used as the temp directory, unless overridden by withTempDirectory(java.lang.String).
  - to
```
public FileIO.Write<DestinationT,UserT> to(ValueProvider<java.lang.String> directory)
```
    Like to(String) but with a ValueProvider.
  - withPrefix
```
public FileIO.Write<DestinationT,UserT> withPrefix(java.lang.String prefix)
```
    Specifies a common prefix to use for all generated filenames, if using the default file naming. Incompatible with withNaming(org.apache.beam.sdk.io.FileIO.Write.FileNaming).
  - withPrefix
```
public FileIO.Write<DestinationT,UserT> withPrefix(ValueProvider<java.lang.String> prefix)
```
    Like withPrefix(String) but with a ValueProvider.
  - withSuffix
```
public FileIO.Write<DestinationT,UserT> withSuffix(java.lang.String suffix)
```
    Specifies a common suffix to use for all generated filenames, if using the default file naming. Incompatible with withNaming(org.apache.beam.sdk.io.FileIO.Write.FileNaming).
  - withSuffix
```
public FileIO.Write<DestinationT,UserT> withSuffix(ValueProvider<java.lang.String> suffix)
```
    Like withSuffix(String) but with a ValueProvider.
  - withNaming
```
public FileIO.Write<DestinationT,UserT> withNaming(FileIO.Write.FileNaming naming)
```
    Specifies a custom strategy for generating filenames. All generated filenames will be resolved relative to the directory specified in to(java.lang.String), if any.
    Incompatible with withSuffix(java.lang.String).
    This can only be used in combination with FileIO.write() but not FileIO.writeDynamic().
  - withNaming
```
public FileIO.Write<DestinationT,UserT> withNaming(SerializableFunction<DestinationT,FileIO.Write.FileNaming> namingFn)
```
    Specifies a custom strategy for generating filenames depending on the destination, similar to withNaming(FileNaming).
    This can only be used in combination with FileIO.writeDynamic() but not FileIO.write().
  - withNaming
```
public FileIO.Write<DestinationT,UserT> withNaming(Contextful<Contextful.Fn<DestinationT,FileIO.Write.FileNaming>> namingFn)
```
    Like withNaming(SerializableFunction) but allows accessing context, such as side inputs, from the function.
  - withTempDirectory
```
public FileIO.Write<DestinationT,UserT> withTempDirectory(java.lang.String tempDirectory)
```
    Specifies a directory into which all temporary files will be placed.
  - withTempDirectory
```
public FileIO.Write<DestinationT,UserT> withTempDirectory(ValueProvider<java.lang.String> tempDirectory)
```
    Like withTempDirectory(String).
  - withCompression
```
public FileIO.Write<DestinationT,UserT> withCompression(Compression compression)
```
    Specifies to compress all generated shard files using the given Compression and, by default, append the respective extension to the filename.
  - withEmptyGlobalWindowDestination
```
public FileIO.Write<DestinationT,UserT> withEmptyGlobalWindowDestination(DestinationT emptyWindowDestination)
```
    If withIgnoreWindowing() is specified, specifies a destination to be used in case the collection is empty, to generate the (only, empty) output file.
  - withDestinationCoder
```
public FileIO.Write<DestinationT,UserT> withDestinationCoder(Coder<DestinationT> destinationCoder)
```
    Specifies a Coder for the destination type, if it can not be inferred from by(org.apache.beam.sdk.transforms.SerializableFunction<UserT, DestinationT>).
  - withNumShards
```
public FileIO.Write<DestinationT,UserT> withNumShards(int numShards)
```
    Specifies to use a given fixed number of shards per window. 0 means runner-determined sharding. Specifying a non-zero value may hurt performance, because it will limit the parallelism of writing and will introduce an extra GroupByKey operation.
  - withNumShards
```
public FileIO.Write<DestinationT,UserT> withNumShards(@Nullable
                                                      ValueProvider<java.lang.Integer> numShards)
```
    Like withNumShards(int). Specifying null means runner-determined sharding.
  - withSharding
```
public FileIO.Write<DestinationT,UserT> withSharding(PTransform<PCollection<UserT>,PCollectionView<java.lang.Integer>> sharding)
```
    Specifies a PTransform to use for computing the desired number of shards in each window.
  - withIgnoreWindowing
```
@Deprecated
public FileIO.Write<DestinationT,UserT> withIgnoreWindowing()
```
    Deprecated. Avoid usage of this method: its effects are complex and it will be removed in future versions of Beam. Right now it exists for compatibility with WriteFiles.
    
    Specifies to ignore windowing information in the input, and instead rewindow it to global window with the default trigger.
  - expand
```
public WriteFilesResult<DestinationT> expand(PCollection<UserT> input)
```
    Description copied from class: PTransform
    
    Override this method to specify how this PTransform should be expanded on the given InputT.
    NOTE: This method should not be called directly. Instead apply the PTransform should be applied to the InputT using the apply method.
    Composite transforms, which are defined in terms of other transforms, should return the output of one of the composed transforms. Non-composite transforms, which do not apply any transforms internally, should return a new unbound output and register evaluators (via backend-specific registration methods).
    
    Specified by:
    
    expand in class PTransform<PCollection<UserT>,WriteFilesResult<DestinationT>>

Class FileIO.Write<DestinationT,UserT>

Nested Class Summary

Field Summary

Fields inherited from class org.apache.beam.sdk.transforms.PTransform

Constructor Summary

Method Summary

Methods inherited from class org.apache.beam.sdk.transforms.PTransform

Methods inherited from class java.lang.Object

Constructor Detail

Write

Method Detail

defaultNaming

defaultNaming

relativeFileNaming

by

by

via

via

via

via

to

to

withPrefix

withPrefix

withSuffix

withSuffix

withNaming

withNaming

withNaming

withTempDirectory

withTempDirectory

withCompression

withEmptyGlobalWindowDestination

withDestinationCoder

withNumShards

withNumShards

withSharding

withIgnoreWindowing

expand