org.apache.beam.runners.dataflow.util

Class GcsStager

java.lang.Object

org.apache.beam.runners.dataflow.util.GcsStager

All Implemented Interfaces:

Stager

public class GcsStager
extends java.lang.Object
implements Stager

Utility class for staging files to GCS.

Method Summary

Modifier and Type	Method and Description
static GcsStager	fromOptions(PipelineOptions options)
java.util.List<com.google.api.services.dataflow.model.DataflowPackage>	stageDefaultFiles() Stages DataflowPipelineWorkerPoolOptions.getFilesToStage(), which defaults to every file on the classpath unless overridden, as well as DataflowPipelineDebugOptions.getOverrideWindmillBinary() if specified.
java.util.List<com.google.api.services.dataflow.model.DataflowPackage>	stageFiles(java.util.List<java.lang.String> filesToStage) Stages files to DataflowPipelineOptions.getStagingLocation(), suffixed with their md5 hash to avoid collisions.
com.google.api.services.dataflow.model.DataflowPackage	stageToFile(byte[] bytes, java.lang.String baseName) Stage bytes to a target file name wherever this stager stages things.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

fromOptions

public static GcsStager fromOptions(PipelineOptions options)

stageDefaultFiles

public java.util.List<com.google.api.services.dataflow.model.DataflowPackage> stageDefaultFiles()

Stages DataflowPipelineWorkerPoolOptions.getFilesToStage(), which defaults to every file on the classpath unless overridden, as well as DataflowPipelineDebugOptions.getOverrideWindmillBinary() if specified.

Specified by:

stageDefaultFiles in interface Stager

See Also:

stageFiles(List)

stageFiles

public java.util.List<com.google.api.services.dataflow.model.DataflowPackage> stageFiles(java.util.List<java.lang.String> filesToStage)

Stages files to DataflowPipelineOptions.getStagingLocation(), suffixed with their md5 hash to avoid collisions.

Uses GcsOptions.getGcsUploadBufferSizeBytes().

Specified by:

stageFiles in interface Stager

stageToFile

public com.google.api.services.dataflow.model.DataflowPackage stageToFile(byte[] bytes,
                                                                          java.lang.String baseName)

Description copied from interface: Stager

Stage bytes to a target file name wherever this stager stages things.

Specified by:

stageToFile in interface Stager