org.apache.beam.sdk.io

Class DefaultFilenamePolicy

java.lang.Object

org.apache.beam.sdk.io.FileBasedSink.FilenamePolicy

org.apache.beam.sdk.io.DefaultFilenamePolicy

All Implemented Interfaces:

java.io.Serializable

public final class DefaultFilenamePolicy
extends FileBasedSink.FilenamePolicy

A default FileBasedSink.FilenamePolicy for windowed and unwindowed files. This policy is constructed using three parameters that together define the output name of a sharded file, in conjunction with the number of shards, index of the particular file, current window and pane information, using constructName(org.apache.beam.sdk.io.fs.ResourceId, java.lang.String, java.lang.String, int, int, java.lang.String, java.lang.String).

Most users will use this DefaultFilenamePolicy. For more advanced uses in generating different files for each window and other sharding controls, see the WriteOneFilePerWindow example pipeline.

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	DefaultFilenamePolicy.Params Encapsulates constructor parameters to DefaultFilenamePolicy.
static class	DefaultFilenamePolicy.ParamsCoder A Coder for DefaultFilenamePolicy.Params.

Field Summary

Fields

Modifier and Type	Field and Description
static java.lang.String	DEFAULT_UNWINDOWED_SHARD_TEMPLATE The default sharding name template.
static java.lang.String	DEFAULT_WINDOWED_SHARD_TEMPLATE The default windowed sharding name template used when writing windowed files.

Method Summary

Modifier and Type	Method and Description
static DefaultFilenamePolicy	fromParams(DefaultFilenamePolicy.Params params) Construct a DefaultFilenamePolicy from a DefaultFilenamePolicy.Params object.
static DefaultFilenamePolicy	fromStandardParameters(ValueProvider<ResourceId> baseFilename, @Nullable java.lang.String shardTemplate, @Nullable java.lang.String filenameSuffix, boolean windowedWrites) Construct a DefaultFilenamePolicy.
void	populateDisplayData(DisplayData.Builder builder) Populates the display data.
@Nullable ResourceId	unwindowedFilename(int shardNumber, int numShards, FileBasedSink.OutputFileHints outputFileHints) When a sink has not requested windowed or triggered output, this method will be invoked to return the file resource to be created given the base output directory and a FileBasedSink.OutputFileHints containing information about the file, including a suggested (e.g.
ResourceId	windowedFilename(int shardNumber, int numShards, BoundedWindow window, PaneInfo paneInfo, FileBasedSink.OutputFileHints outputFileHints) When a sink has requested windowed or triggered output, this method will be invoked to return the file resource to be created given the base output directory and a FileBasedSink.OutputFileHints containing information about the file, including a suggested extension (e.g.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

DEFAULT_UNWINDOWED_SHARD_TEMPLATE

public static final java.lang.String DEFAULT_UNWINDOWED_SHARD_TEMPLATE

The default sharding name template.

See Also:

Constant Field Values

DEFAULT_WINDOWED_SHARD_TEMPLATE

public static final java.lang.String DEFAULT_WINDOWED_SHARD_TEMPLATE

The default windowed sharding name template used when writing windowed files. This is used as default in cases when user did not specify shard template to be used and there is a need to write windowed files. In cases when user does specify shard template to be used then provided template will be used for both windowed and non-windowed file names.

See Also:

Constant Field Values

Method Detail

fromStandardParameters

public static DefaultFilenamePolicy fromStandardParameters(ValueProvider<ResourceId> baseFilename,
                                                           @Nullable java.lang.String shardTemplate,
                                                           @Nullable java.lang.String filenameSuffix,
                                                           boolean windowedWrites)

Construct a DefaultFilenamePolicy.

This is a shortcut for:

 DefaultFilenamePolicy.fromParams(new Params()
   .withBaseFilename(baseFilename)
   .withShardTemplate(shardTemplate)
   .withSuffix(filenameSuffix)
   .withWindowedWrites())
 

Where the respective with methods are invoked only if the value is non-null or true.

fromParams

public static DefaultFilenamePolicy fromParams(DefaultFilenamePolicy.Params params)

Construct a DefaultFilenamePolicy from a DefaultFilenamePolicy.Params object.

unwindowedFilename

public @Nullable ResourceId unwindowedFilename(int shardNumber,
                                               int numShards,
                                               FileBasedSink.OutputFileHints outputFileHints)

Description copied from class: FileBasedSink.FilenamePolicy

When a sink has not requested windowed or triggered output, this method will be invoked to return the file resource to be created given the base output directory and a FileBasedSink.OutputFileHints containing information about the file, including a suggested (e.g. coming from Compression).

The shardNumber and numShards parameters, should be used by the policy to generate unique and consistent filenames.

Specified by:

unwindowedFilename in class FileBasedSink.FilenamePolicy

windowedFilename

public ResourceId windowedFilename(int shardNumber,
                                   int numShards,
                                   BoundedWindow window,
                                   PaneInfo paneInfo,
                                   FileBasedSink.OutputFileHints outputFileHints)

Description copied from class: FileBasedSink.FilenamePolicy

When a sink has requested windowed or triggered output, this method will be invoked to return the file resource to be created given the base output directory and a FileBasedSink.OutputFileHints containing information about the file, including a suggested extension (e.g. coming from Compression).

The policy must return unique and consistent filenames for different windows and panes.

Specified by:

windowedFilename in class FileBasedSink.FilenamePolicy

populateDisplayData

public void populateDisplayData(DisplayData.Builder builder)

Description copied from class: FileBasedSink.FilenamePolicy

Populates the display data.

Overrides:

populateDisplayData in class FileBasedSink.FilenamePolicy