FileBasedSink.WriteOperation (Apache Beam 2.27.0)

java.lang.Object
- org.apache.beam.sdk.io.FileBasedSink.WriteOperation<DestinationT,OutputT>

Type Parameters:

OutputT - the type of values written to the sink.

All Implemented Interfaces:

java.io.Serializable

Enclosing class:

FileBasedSink<UserT,DestinationT,OutputT>
```
public abstract static class FileBasedSink.WriteOperation<DestinationT,OutputT>
extends java.lang.Object
implements java.io.Serializable
```
Abstract operation that manages the process of writing to FileBasedSink.
The primary responsibilities of the WriteOperation is the management of output files. During a write, FileBasedSink.Writers write bundles to temporary file locations. After the bundles have been written,
1. finalizeDestination(DestinationT, org.apache.beam.sdk.transforms.windowing.BoundedWindow, java.lang.Integer, java.util.Collection<org.apache.beam.sdk.io.FileBasedSink.FileResult<DestinationT>>) is given a list of the temporary files containing the output bundles.
2. During finalize, these temporary files are copied to final output locations and named according to a file naming template.
3. Finally, any temporary files that were created during the write are removed.
Subclass implementations of WriteOperation must implement createWriter() to return a concrete FileBasedSinkWriter.
Temporary and Output File Naming:

During the write, bundles are written to temporary files using the tempDirectory that can be provided via the constructor of WriteOperation. These temporary files will be named {tempDirectory}/{bundleId}, where bundleId is the unique id of the bundle. For example, if tempDirectory is "gs://my-bucket/my_temp_output", the output for a bundle with bundle id 15723 will be "gs://my-bucket/my_temp_output/15723".
Final output files are written to the location specified by the FileBasedSink.FilenamePolicy. If no filename policy is specified, then the DefaultFilenamePolicy will be used. The directory that the files are written to is determined by the FileBasedSink.FilenamePolicy instance.
Note that in the case of permanent failure of a bundle's write, no clean up of temporary files will occur.
If there are no elements in the PCollection being written, no output will be generated.
See Also:

Serialized Form

Field Summary

Fields
Modifier and Type	Field and Description
`protected FileBasedSink<?,DestinationT,OutputT>`	`sink` The Sink that this WriteOperation will write to.
`protected ValueProvider<ResourceId>`	`tempDirectory` Directory for temporary output files.
`protected boolean`	`windowedWrites` Whether windowed writes are being used.

Constructor Summary

Constructors
Constructor and Description
`WriteOperation(FileBasedSink<?,DestinationT,OutputT> sink)` Constructs a WriteOperation using the default strategy for generating a temporary directory from the base output filename.
`WriteOperation(FileBasedSink<?,DestinationT,OutputT> sink, ResourceId tempDirectory)` Create a new WriteOperation.

Method Summary

All Methods Static Methods Instance Methods Abstract Methods Concrete Methods
Modifier and Type	Method and Description
`protected static ResourceId`	`buildTemporaryFilename(ResourceId tempDirectory, java.lang.String filename)` Constructs a temporary file resource given the temporary directory and a filename.
`abstract FileBasedSink.Writer<DestinationT,OutputT>`	`createWriter()` Clients must implement to return a subclass of `FileBasedSink.Writer`.
`protected java.util.List<KV<FileBasedSink.FileResult<DestinationT>,ResourceId>>`	`finalizeDestination(DestinationT dest, @Nullable BoundedWindow window, @Nullable java.lang.Integer numShards, java.util.Collection<FileBasedSink.FileResult<DestinationT>> existingResults)`
`FileBasedSink<?,DestinationT,OutputT>`	`getSink()` Returns the FileBasedSink for this write operation.
`void`	`removeTemporaryFiles(java.util.Collection<ResourceId> filenames)`
`void`	`setWindowedWrites(boolean windowedWrites)` Indicates that the operation will be performing windowed writes.
`java.lang.String`	`toString()`

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

sink

protected final FileBasedSink<?,DestinationT,OutputT> sink

The Sink that this WriteOperation will write to.

tempDirectory

protected final ValueProvider<ResourceId> tempDirectory

Directory for temporary output files.

windowedWrites

@Experimental(value=FILESYSTEM)
protected boolean windowedWrites

Whether windowed writes are being used.

Constructor Detail
- WriteOperation
```
public WriteOperation(FileBasedSink<?,DestinationT,OutputT> sink)
```
  Constructs a WriteOperation using the default strategy for generating a temporary directory from the base output filename.
  Default is a uniquely named subdirectory of the provided tempDirectory, e.g. if tempDirectory is /path/to/foo/, the temporary directory will be /path/to/foo/.temp-beam-$uuid.
  
  Parameters:
  
  sink - the FileBasedSink that will be used to configure this write operation.
- WriteOperation
```
@Experimental(value=FILESYSTEM)
public WriteOperation(FileBasedSink<?,DestinationT,OutputT> sink,
                                                      ResourceId tempDirectory)
```
  Create a new WriteOperation.
  
  Parameters:
  
  sink - the FileBasedSink that will be used to configure this write operation.
  
  tempDirectory - the base directory to be used for temporary output files.

Method Detail

buildTemporaryFilename

@Experimental(value=FILESYSTEM)
protected static ResourceId buildTemporaryFilename(ResourceId tempDirectory,
                                                                                   java.lang.String filename)
                                                                            throws java.io.IOException

Constructs a temporary file resource given the temporary directory and a filename.

Throws:: java.io.IOException

createWriter

public abstract FileBasedSink.Writer<DestinationT,OutputT> createWriter()
                                                                 throws java.lang.Exception

Clients must implement to return a subclass of FileBasedSink.Writer. This method must not mutate the state of the object.

Throws:: java.lang.Exception

setWindowedWrites
```
public void setWindowedWrites(boolean windowedWrites)
```
Indicates that the operation will be performing windowed writes.

removeTemporaryFiles

public void removeTemporaryFiles(java.util.Collection<ResourceId> filenames)
                          throws java.io.IOException

Throws:: java.io.IOException

finalizeDestination

@Experimental(value=FILESYSTEM)
protected final java.util.List<KV<FileBasedSink.FileResult<DestinationT>,ResourceId>> finalizeDestination(DestinationT dest,
                                                                                                                                          @Nullable BoundedWindow window,
                                                                                                                                          @Nullable java.lang.Integer numShards,
                                                                                                                                          java.util.Collection<FileBasedSink.FileResult<DestinationT>> existingResults)
                                                                                                                                   throws java.lang.Exception

Throws:: java.lang.Exception

getSink

public FileBasedSink<?,DestinationT,OutputT> getSink()

Returns the FileBasedSink for this write operation.

toString
```
public java.lang.String toString()
```
Overrides:

toString in class java.lang.Object

Class FileBasedSink.WriteOperation<DestinationT,OutputT>

Temporary and Output File Naming: