org.apache.beam.sdk.io

Class DefaultFilenamePolicy

java.lang.Object

org.apache.beam.sdk.io.FileBasedSink.FilenamePolicy

org.apache.beam.sdk.io.DefaultFilenamePolicy

All Implemented Interfaces:

java.io.Serializable

public final class DefaultFilenamePolicy
extends FileBasedSink.FilenamePolicy

A default FileBasedSink.FilenamePolicy for windowed and unwindowed files. This policy is constructed using three parameters that together define the output name of a sharded file, in conjunction with the number of shards, index of the particular file, current window and pane information, using constructName(org.apache.beam.sdk.io.fs.ResourceId, java.lang.String, java.lang.String, int, int, java.lang.String, java.lang.String).

Most users will use this DefaultFilenamePolicy. For more advanced uses in generating different files for each window and other sharding controls, see the WriteOneFilePerWindow example pipeline.

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	DefaultFilenamePolicy.Params Encapsulates constructor parameters to DefaultFilenamePolicy.
static class	DefaultFilenamePolicy.ParamsCoder A Coder for DefaultFilenamePolicy.Params.

Field Summary

Fields

Modifier and Type	Field and Description
static java.lang.String	DEFAULT_UNWINDOWED_SHARD_TEMPLATE The default sharding name template.

Method Summary

Modifier and Type	Method and Description
static DefaultFilenamePolicy	fromParams(DefaultFilenamePolicy.Params params) Construct a DefaultFilenamePolicy from a DefaultFilenamePolicy.Params object.
static DefaultFilenamePolicy	fromStandardParameters(ValueProvider<ResourceId> baseFilename, java.lang.String shardTemplate, java.lang.String filenameSuffix, boolean windowedWrites) Construct a DefaultFilenamePolicy.
void	populateDisplayData(DisplayData.Builder builder) Populates the display data.
ResourceId	unwindowedFilename(int shardNumber, int numShards, FileBasedSink.OutputFileHints outputFileHints) When a sink has not requested windowed or triggered output, this method will be invoked to return the file resource to be created given the base output directory and a FileBasedSink.OutputFileHints containing information about the file, including a suggested (e.g.
ResourceId	windowedFilename(int shardNumber, int numShards, BoundedWindow window, PaneInfo paneInfo, FileBasedSink.OutputFileHints outputFileHints) When a sink has requested windowed or triggered output, this method will be invoked to return the file resource to be created given the base output directory and a FileBasedSink.OutputFileHints containing information about the file, including a suggested extension (e.g.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

DEFAULT_UNWINDOWED_SHARD_TEMPLATE

public static final java.lang.String DEFAULT_UNWINDOWED_SHARD_TEMPLATE

The default sharding name template.

See Also:

Constant Field Values

Method Detail

fromStandardParameters

public static DefaultFilenamePolicy fromStandardParameters(ValueProvider<ResourceId> baseFilename,
                                                           @Nullable
                                                           java.lang.String shardTemplate,
                                                           @Nullable
                                                           java.lang.String filenameSuffix,
                                                           boolean windowedWrites)

Construct a DefaultFilenamePolicy.

This is a shortcut for:

 DefaultFilenamePolicy.fromParams(new Params()
   .withBaseFilename(baseFilename)
   .withShardTemplate(shardTemplate)
   .withSuffix(filenameSuffix)
   .withWindowedWrites())
 

Where the respective with methods are invoked only if the value is non-null or true.

fromParams

public static DefaultFilenamePolicy fromParams(DefaultFilenamePolicy.Params params)

Construct a DefaultFilenamePolicy from a DefaultFilenamePolicy.Params object.

unwindowedFilename

@Nullable
public ResourceId unwindowedFilename(int shardNumber,
                                               int numShards,
                                               FileBasedSink.OutputFileHints outputFileHints)

Description copied from class: FileBasedSink.FilenamePolicy

When a sink has not requested windowed or triggered output, this method will be invoked to return the file resource to be created given the base output directory and a FileBasedSink.OutputFileHints containing information about the file, including a suggested (e.g. coming from Compression).

The shardNumber and numShards parameters, should be used by the policy to generate unique and consistent filenames.

Specified by:

unwindowedFilename in class FileBasedSink.FilenamePolicy

windowedFilename

public ResourceId windowedFilename(int shardNumber,
                                   int numShards,
                                   BoundedWindow window,
                                   PaneInfo paneInfo,
                                   FileBasedSink.OutputFileHints outputFileHints)

Description copied from class: FileBasedSink.FilenamePolicy

When a sink has requested windowed or triggered output, this method will be invoked to return the file resource to be created given the base output directory and a FileBasedSink.OutputFileHints containing information about the file, including a suggested extension (e.g. coming from Compression).

The policy must return unique and consistent filenames for different windows and panes.

Specified by:

windowedFilename in class FileBasedSink.FilenamePolicy

populateDisplayData

public void populateDisplayData(DisplayData.Builder builder)

Description copied from class: FileBasedSink.FilenamePolicy

Populates the display data.

Overrides:

populateDisplayData in class FileBasedSink.FilenamePolicy