org.apache.beam.runners.dataflow.util

Class GcsStager

java.lang.Object

org.apache.beam.runners.dataflow.util.GcsStager

All Implemented Interfaces:

Stager

public class GcsStager
extends java.lang.Object
implements Stager

Utility class for staging files to GCS.

Method Summary

Modifier and Type	Method and Description
static GcsStager	fromOptions(PipelineOptions options)
java.util.List<DataflowPackage>	stageFiles(java.util.List<PackageUtil.StagedFile> filesToStage) Stages files to DataflowPipelineOptions.getStagingLocation(), suffixed with their md5 hash to avoid collisions.
DataflowPackage	stageToFile(byte[] bytes, java.lang.String baseName) Stage bytes to a target file name wherever this stager stages things.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

fromOptions

public static GcsStager fromOptions(PipelineOptions options)

stageFiles

public java.util.List<DataflowPackage> stageFiles(java.util.List<PackageUtil.StagedFile> filesToStage)

Stages files to DataflowPipelineOptions.getStagingLocation(), suffixed with their md5 hash to avoid collisions.

Uses GcsOptions.getGcsUploadBufferSizeBytes().

Specified by:

stageFiles in interface Stager

stageToFile

public DataflowPackage stageToFile(byte[] bytes,
                                   java.lang.String baseName)

Description copied from interface: Stager

Stage bytes to a target file name wherever this stager stages things.

Specified by:

stageToFile in interface Stager