org.apache.beam.sdk.io

Class DefaultFilenamePolicy

java.lang.Object

org.apache.beam.sdk.io.FileBasedSink.FilenamePolicy

org.apache.beam.sdk.io.DefaultFilenamePolicy

All Implemented Interfaces:

java.io.Serializable

public final class DefaultFilenamePolicy
extends FileBasedSink.FilenamePolicy

A default FileBasedSink.FilenamePolicy for windowed and unwindowed files. This policy is constructed using three parameters that together define the output name of a sharded file, in conjunction with the number of shards, index of the particular file, current window and pane information, using constructName(java.lang.String, java.lang.String, java.lang.String, int, int, java.lang.String, java.lang.String).

Most users will use this DefaultFilenamePolicy. For more advanced uses in generating different files for each window and other sharding controls, see the WriteOneFilePerWindow example pipeline.

See Also:

Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.beam.sdk.io.FileBasedSink.FilenamePolicy

FileBasedSink.FilenamePolicy.Context, FileBasedSink.FilenamePolicy.WindowedContext

Field Summary

Fields

Modifier and Type	Field and Description
static java.lang.String	DEFAULT_UNWINDOWED_SHARD_TEMPLATE The default sharding name template used in constructUsingStandardParameters(org.apache.beam.sdk.options.ValueProvider<org.apache.beam.sdk.io.fs.ResourceId>, java.lang.String, java.lang.String, boolean).

Method Summary

Modifier and Type	Method and Description
static DefaultFilenamePolicy	constructUsingStandardParameters(ValueProvider<ResourceId> outputPrefix, java.lang.String shardTemplate, java.lang.String filenameSuffix, boolean windowedWrites) A helper function to construct a DefaultFilenamePolicy using the standard filename parameters, namely a provided ResourceId for the output prefix, and possibly-null shard name template and suffix.
void	populateDisplayData(DisplayData.Builder builder) Populates the display data.
ResourceId	unwindowedFilename(ResourceId outputDirectory, FileBasedSink.FilenamePolicy.Context context, java.lang.String extension) When a sink has not requested windowed or triggered output, this method will be invoked to return the file resource to be created given the base output directory and a (possibly empty) extension applied by additional FileBasedSink configuration (e.g., FileBasedSink.CompressionType).
ResourceId	windowedFilename(ResourceId outputDirectory, FileBasedSink.FilenamePolicy.WindowedContext context, java.lang.String extension) When a sink has requested windowed or triggered output, this method will be invoked to return the file resource to be created given the base output directory and a (possibly empty) extension from FileBasedSink configuration (e.g., FileBasedSink.CompressionType).

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

DEFAULT_UNWINDOWED_SHARD_TEMPLATE

public static final java.lang.String DEFAULT_UNWINDOWED_SHARD_TEMPLATE

The default sharding name template used in constructUsingStandardParameters(org.apache.beam.sdk.options.ValueProvider<org.apache.beam.sdk.io.fs.ResourceId>, java.lang.String, java.lang.String, boolean).

See Also:

Constant Field Values

Method Detail

constructUsingStandardParameters

public static DefaultFilenamePolicy constructUsingStandardParameters(ValueProvider<ResourceId> outputPrefix,
                                                                     @Nullable
                                                                     java.lang.String shardTemplate,
                                                                     @Nullable
                                                                     java.lang.String filenameSuffix,
                                                                     boolean windowedWrites)

A helper function to construct a DefaultFilenamePolicy using the standard filename parameters, namely a provided ResourceId for the output prefix, and possibly-null shard name template and suffix.

Any filename component of the provided resource will be used as the filename prefix.

If provided, the shard name template will be used; otherwise DEFAULT_UNWINDOWED_SHARD_TEMPLATE will be used for non-windowed file names and DEFAULT_WINDOWED_SHARD_TEMPLATE will be used for windowed file names.

If provided, the suffix will be used; otherwise the files will have an empty suffix.

unwindowedFilename

@Nullable
public ResourceId unwindowedFilename(ResourceId outputDirectory,
                                               FileBasedSink.FilenamePolicy.Context context,
                                               java.lang.String extension)

Description copied from class: FileBasedSink.FilenamePolicy

When a sink has not requested windowed or triggered output, this method will be invoked to return the file resource to be created given the base output directory and a (possibly empty) extension applied by additional FileBasedSink configuration (e.g., FileBasedSink.CompressionType).

The FileBasedSink.FilenamePolicy.Context object only provides sharding information, which is used by the policy to generate unique and consistent filenames.

Specified by:

unwindowedFilename in class FileBasedSink.FilenamePolicy

windowedFilename

public ResourceId windowedFilename(ResourceId outputDirectory,
                                   FileBasedSink.FilenamePolicy.WindowedContext context,
                                   java.lang.String extension)

Description copied from class: FileBasedSink.FilenamePolicy

When a sink has requested windowed or triggered output, this method will be invoked to return the file resource to be created given the base output directory and a (possibly empty) extension from FileBasedSink configuration (e.g., FileBasedSink.CompressionType).

The FileBasedSink.FilenamePolicy.WindowedContext object gives access to the window and pane, as well as sharding information. The policy must return unique and consistent filenames for different windows and panes.

Specified by:

windowedFilename in class FileBasedSink.FilenamePolicy

populateDisplayData

public void populateDisplayData(DisplayData.Builder builder)

Description copied from class: FileBasedSink.FilenamePolicy

Populates the display data.

Overrides:

populateDisplayData in class FileBasedSink.FilenamePolicy