org.apache.beam.runners.flink

Interface FlinkPipelineOptions

All Superinterfaces:

ApplicationNameOptions, HasDisplayData, PipelineOptions, StreamingOptions

public interface FlinkPipelineOptions
extends PipelineOptions, ApplicationNameOptions, StreamingOptions

Options which can be used to configure a Flink PortablePipelineRunner.

Avoid using `org.apache.flink.*` members below. This allows including the flink runner without requiring flink on the classpath (e.g. to use with the direct runner).

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.beam.sdk.options.PipelineOptions

PipelineOptions.AtomicLongFactory, PipelineOptions.CheckEnabled, PipelineOptions.DirectRunner, PipelineOptions.JobNameFactory, PipelineOptions.UserAgentFactory

Field Summary

Fields

Modifier and Type	Field and Description
static java.lang.String	AUTO
static java.lang.String	EXACTLY_ONCE
static java.lang.String	PIPELINED

Method Summary

Modifier and Type	Method and Description
java.lang.Boolean	getAllowNonRestoredState()
java.lang.Long	getAutoWatermarkInterval()
java.lang.Long	getCheckpointingInterval()
java.lang.String	getCheckpointingMode()
java.lang.Long	getCheckpointTimeoutMillis()
java.lang.Boolean	getEnableMetrics()
java.lang.String	getExecutionModeForBatch()
java.lang.Long	getExecutionRetryDelay()
java.lang.Boolean	getFailOnCheckpointingErrors()
java.util.List<java.lang.String>	getFilesToStage() List of local files to make available to workers.
java.lang.String	getFlinkMaster() The url of the Flink JobManager on which to execute pipelines.
java.lang.Long	getLatencyTrackingInterval()
java.lang.Long	getMaxBundleSize()
java.lang.Long	getMaxBundleTimeMills()
java.lang.Integer	getMaxParallelism()
java.lang.Long	getMinPauseBetweenCheckpoints()
java.lang.Integer	getNumberOfExecutionRetries()
java.lang.Boolean	getObjectReuse()
java.lang.Integer	getParallelism()
java.lang.Boolean	getRetainExternalizedCheckpointsOnCancellation()
java.lang.String	getSavepointPath()
java.lang.Class<? extends FlinkStateBackendFactory>	getStateBackendFactory() State backend to store Beam's state during computation.
java.lang.Boolean	isAutoBalanceWriteFilesShardingEnabled()
java.lang.Boolean	isExternalizedCheckpointsEnabled() Enables or disables externalized checkpoints.
java.lang.Boolean	isShutdownSourcesOnFinalWatermark() Whether to shutdown sources when their watermark reaches +Inf.
void	setAllowNonRestoredState(java.lang.Boolean allowNonRestoredState)
void	setAutoBalanceWriteFilesShardingEnabled(java.lang.Boolean autoBalanceWriteFilesShardingEnabled)
void	setAutoWatermarkInterval(java.lang.Long interval)
void	setCheckpointingInterval(java.lang.Long interval)
void	setCheckpointingMode(java.lang.String mode)
void	setCheckpointTimeoutMillis(java.lang.Long checkpointTimeoutMillis)
void	setEnableMetrics(java.lang.Boolean enableMetrics)
void	setExecutionModeForBatch(java.lang.String executionMode)
void	setExecutionRetryDelay(java.lang.Long delay)
void	setExternalizedCheckpointsEnabled(java.lang.Boolean externalCheckpoints)
void	setFailOnCheckpointingErrors(java.lang.Boolean failOnCheckpointingErrors)
void	setFilesToStage(java.util.List<java.lang.String> value)
void	setFlinkMaster(java.lang.String value)
void	setLatencyTrackingInterval(java.lang.Long interval)
void	setMaxBundleSize(java.lang.Long size)
void	setMaxBundleTimeMills(java.lang.Long time)
void	setMaxParallelism(java.lang.Integer value)
void	setMinPauseBetweenCheckpoints(java.lang.Long minPauseInterval)
void	setNumberOfExecutionRetries(java.lang.Integer retries)
void	setObjectReuse(java.lang.Boolean reuse)
void	setParallelism(java.lang.Integer value)
void	setRetainExternalizedCheckpointsOnCancellation(java.lang.Boolean retainOnCancellation)
void	setSavepointPath(java.lang.String path)
void	setShutdownSourcesOnFinalWatermark(java.lang.Boolean shutdownOnFinalWatermark)
void	setStateBackendFactory(java.lang.Class<? extends FlinkStateBackendFactory> stateBackendFactory)

Methods inherited from interface org.apache.beam.sdk.options.StreamingOptions

isStreaming, setStreaming

Methods inherited from interface org.apache.beam.sdk.options.ApplicationNameOptions

getAppName, setAppName

Methods inherited from interface org.apache.beam.sdk.options.PipelineOptions

as, getJobName, getOptionsId, getRunner, getStableUniqueNames, getTempLocation, getUserAgent, outputRuntimeOptions, setJobName, setOptionsId, setRunner, setStableUniqueNames, setTempLocation, setUserAgent

Methods inherited from interface org.apache.beam.sdk.transforms.display.HasDisplayData

populateDisplayData

Field Detail

AUTO

static final java.lang.String AUTO

See Also:

Constant Field Values

PIPELINED

static final java.lang.String PIPELINED

See Also:

Constant Field Values

EXACTLY_ONCE

static final java.lang.String EXACTLY_ONCE

See Also:

Constant Field Values

Method Detail

getFilesToStage

java.util.List<java.lang.String> getFilesToStage()

List of local files to make available to workers.

Jars are placed on the worker's classpath.

The default value is the list of jars from the main program's classpath.

setFilesToStage

void setFilesToStage(java.util.List<java.lang.String> value)

getFlinkMaster

@Default.String(value="[auto]")
java.lang.String getFlinkMaster()

The url of the Flink JobManager on which to execute pipelines. This can either be the the address of a cluster JobManager, in the form "host:port" or one of the special Strings "[local]", "[collection]" or "[auto]". "[local]" will start a local Flink Cluster in the JVM, "[collection]" will execute the pipeline on Java Collections while "[auto]" will let the system decide where to execute the pipeline based on the environment.

setFlinkMaster

void setFlinkMaster(java.lang.String value)

getParallelism

@Default.Integer(value=-1)
java.lang.Integer getParallelism()

setParallelism

void setParallelism(java.lang.Integer value)

getMaxParallelism

@Default.Integer(value=-1)
java.lang.Integer getMaxParallelism()

setMaxParallelism

void setMaxParallelism(java.lang.Integer value)

getCheckpointingInterval

@Default.Long(value=-1L)
java.lang.Long getCheckpointingInterval()

setCheckpointingInterval

void setCheckpointingInterval(java.lang.Long interval)

getCheckpointingMode

@Default.String(value="EXACTLY_ONCE")
java.lang.String getCheckpointingMode()

setCheckpointingMode

void setCheckpointingMode(java.lang.String mode)

getCheckpointTimeoutMillis

@Default.Long(value=-1L)
java.lang.Long getCheckpointTimeoutMillis()

setCheckpointTimeoutMillis

void setCheckpointTimeoutMillis(java.lang.Long checkpointTimeoutMillis)

getMinPauseBetweenCheckpoints

@Default.Long(value=-1L)
java.lang.Long getMinPauseBetweenCheckpoints()

setMinPauseBetweenCheckpoints

void setMinPauseBetweenCheckpoints(java.lang.Long minPauseInterval)

getFailOnCheckpointingErrors

@Default.Boolean(value=true)
java.lang.Boolean getFailOnCheckpointingErrors()

setFailOnCheckpointingErrors

void setFailOnCheckpointingErrors(java.lang.Boolean failOnCheckpointingErrors)

getNumberOfExecutionRetries

@Default.Integer(value=-1)
java.lang.Integer getNumberOfExecutionRetries()

setNumberOfExecutionRetries

void setNumberOfExecutionRetries(java.lang.Integer retries)

getExecutionRetryDelay

@Default.Long(value=-1L)
java.lang.Long getExecutionRetryDelay()

setExecutionRetryDelay

void setExecutionRetryDelay(java.lang.Long delay)

getObjectReuse

@Default.Boolean(value=false)
java.lang.Boolean getObjectReuse()

setObjectReuse

void setObjectReuse(java.lang.Boolean reuse)

getStateBackendFactory

java.lang.Class<? extends FlinkStateBackendFactory> getStateBackendFactory()

State backend to store Beam's state during computation. Note: Only applicable when executing in streaming mode.

setStateBackendFactory

void setStateBackendFactory(java.lang.Class<? extends FlinkStateBackendFactory> stateBackendFactory)

getEnableMetrics

@Default.Boolean(value=true)
java.lang.Boolean getEnableMetrics()

setEnableMetrics

void setEnableMetrics(java.lang.Boolean enableMetrics)

isExternalizedCheckpointsEnabled

@Default.Boolean(value=false)
java.lang.Boolean isExternalizedCheckpointsEnabled()

Enables or disables externalized checkpoints.

setExternalizedCheckpointsEnabled

void setExternalizedCheckpointsEnabled(java.lang.Boolean externalCheckpoints)

getRetainExternalizedCheckpointsOnCancellation

@Default.Boolean(value=false)
java.lang.Boolean getRetainExternalizedCheckpointsOnCancellation()

setRetainExternalizedCheckpointsOnCancellation

void setRetainExternalizedCheckpointsOnCancellation(java.lang.Boolean retainOnCancellation)

getMaxBundleSize

@Default.Long(value=1000L)
java.lang.Long getMaxBundleSize()

setMaxBundleSize

void setMaxBundleSize(java.lang.Long size)

getMaxBundleTimeMills

@Default.Long(value=1000L)
java.lang.Long getMaxBundleTimeMills()

setMaxBundleTimeMills

void setMaxBundleTimeMills(java.lang.Long time)

isShutdownSourcesOnFinalWatermark

@Default.Boolean(value=false)
java.lang.Boolean isShutdownSourcesOnFinalWatermark()

Whether to shutdown sources when their watermark reaches +Inf. For production use cases you want this to be disabled because Flink will currently (versions <= 1.5) stop doing checkpoints when any operator (which includes sources) is finished.

Please see FLINK-2491 for progress on this issue.

setShutdownSourcesOnFinalWatermark

void setShutdownSourcesOnFinalWatermark(java.lang.Boolean shutdownOnFinalWatermark)

getLatencyTrackingInterval

@Default.Long(value=0L)
java.lang.Long getLatencyTrackingInterval()

setLatencyTrackingInterval

void setLatencyTrackingInterval(java.lang.Long interval)

getAutoWatermarkInterval

java.lang.Long getAutoWatermarkInterval()

setAutoWatermarkInterval

void setAutoWatermarkInterval(java.lang.Long interval)

getExecutionModeForBatch

@Default.String(value="PIPELINED")
java.lang.String getExecutionModeForBatch()

setExecutionModeForBatch

void setExecutionModeForBatch(java.lang.String executionMode)

getSavepointPath

java.lang.String getSavepointPath()

setSavepointPath

void setSavepointPath(java.lang.String path)

getAllowNonRestoredState

@Default.Boolean(value=false)
java.lang.Boolean getAllowNonRestoredState()

setAllowNonRestoredState

void setAllowNonRestoredState(java.lang.Boolean allowNonRestoredState)

isAutoBalanceWriteFilesShardingEnabled

@Default.Boolean(value=false)
java.lang.Boolean isAutoBalanceWriteFilesShardingEnabled()

setAutoBalanceWriteFilesShardingEnabled

void setAutoBalanceWriteFilesShardingEnabled(java.lang.Boolean autoBalanceWriteFilesShardingEnabled)