org.apache.beam.runners.flink

Interface FlinkPipelineOptions

All Superinterfaces:

ApplicationNameOptions, HasDisplayData, PipelineOptions, StreamingOptions

public interface FlinkPipelineOptions
extends PipelineOptions, ApplicationNameOptions, StreamingOptions

Options which can be used to configure a Flink PipelineRunner.

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.beam.sdk.options.PipelineOptions

PipelineOptions.AtomicLongFactory, PipelineOptions.CheckEnabled, PipelineOptions.DirectRunner, PipelineOptions.JobNameFactory, PipelineOptions.NoOpMetricsSink, PipelineOptions.UserAgentFactory

Method Summary

Modifier and Type	Method and Description
java.lang.Long	getCheckpointingInterval()
org.apache.flink.streaming.api.CheckpointingMode	getCheckpointingMode()
java.lang.Long	getCheckpointTimeoutMillis()
java.lang.Boolean	getEnableMetrics()
java.lang.Long	getExecutionRetryDelay()
java.util.List<java.lang.String>	getFilesToStage() List of local files to make available to workers.
java.lang.String	getFlinkMaster() The url of the Flink JobManager on which to execute pipelines.
java.lang.Long	getMaxBundleSize()
java.lang.Long	getMaxBundleTimeMills()
java.lang.Integer	getNumberOfExecutionRetries()
java.lang.Boolean	getObjectReuse()
java.lang.Integer	getParallelism()
java.lang.Boolean	getRetainExternalizedCheckpointsOnCancellation()
org.apache.flink.runtime.state.AbstractStateBackend	getStateBackend() State backend to store Beam's state during computation.
java.lang.Boolean	isExternalizedCheckpointsEnabled() Enables or disables externalized checkpoints.
java.lang.Boolean	isShutdownSourcesOnFinalWatermark() Whether to shutdown sources when their watermark reaches +Inf.
void	setCheckpointingInterval(java.lang.Long interval)
void	setCheckpointingMode(org.apache.flink.streaming.api.CheckpointingMode mode)
void	setCheckpointTimeoutMillis(java.lang.Long checkpointTimeoutMillis)
void	setEnableMetrics(java.lang.Boolean enableMetrics)
void	setExecutionRetryDelay(java.lang.Long delay)
void	setExternalizedCheckpointsEnabled(java.lang.Boolean externalCheckpoints)
void	setFilesToStage(java.util.List<java.lang.String> value)
void	setFlinkMaster(java.lang.String value)
void	setMaxBundleSize(java.lang.Long size)
void	setMaxBundleTimeMills(java.lang.Long time)
void	setNumberOfExecutionRetries(java.lang.Integer retries)
void	setObjectReuse(java.lang.Boolean reuse)
void	setParallelism(java.lang.Integer value)
void	setRetainExternalizedCheckpointsOnCancellation(java.lang.Boolean retainOnCancellation)
void	setShutdownSourcesOnFinalWatermark(java.lang.Boolean shutdownOnFinalWatermark)
void	setStateBackend(org.apache.flink.runtime.state.AbstractStateBackend stateBackend)

Methods inherited from interface org.apache.beam.sdk.options.StreamingOptions

isStreaming, setStreaming

Methods inherited from interface org.apache.beam.sdk.options.ApplicationNameOptions

getAppName, setAppName

Methods inherited from interface org.apache.beam.sdk.options.PipelineOptions

as, getJobName, getMetricsHttpSinkUrl, getMetricsPushPeriod, getMetricsSink, getOptionsId, getRunner, getStableUniqueNames, getTempLocation, getUserAgent, outputRuntimeOptions, setJobName, setMetricsHttpSinkUrl, setMetricsPushPeriod, setMetricsSink, setOptionsId, setRunner, setStableUniqueNames, setTempLocation, setUserAgent

Methods inherited from interface org.apache.beam.sdk.transforms.display.HasDisplayData

populateDisplayData

Method Detail

getFilesToStage

java.util.List<java.lang.String> getFilesToStage()

List of local files to make available to workers.

Jars are placed on the worker's classpath.

The default value is the list of jars from the main program's classpath.

setFilesToStage

void setFilesToStage(java.util.List<java.lang.String> value)

getFlinkMaster

java.lang.String getFlinkMaster()

The url of the Flink JobManager on which to execute pipelines. This can either be the the address of a cluster JobManager, in the form "host:port" or one of the special Strings "[local]", "[collection]" or "[auto]". "[local]" will start a local Flink Cluster in the JVM, "[collection]" will execute the pipeline on Java Collections while "[auto]" will let the system decide where to execute the pipeline based on the environment.

setFlinkMaster

void setFlinkMaster(java.lang.String value)

getParallelism

@Default.InstanceFactory(value=DefaultParallelismFactory.class)
java.lang.Integer getParallelism()

setParallelism

void setParallelism(java.lang.Integer value)

getCheckpointingInterval

@Default.Long(value=-1L)
java.lang.Long getCheckpointingInterval()

setCheckpointingInterval

void setCheckpointingInterval(java.lang.Long interval)

getCheckpointingMode

@Default.Enum(value="AT_LEAST_ONCE")
org.apache.flink.streaming.api.CheckpointingMode getCheckpointingMode()

setCheckpointingMode

void setCheckpointingMode(org.apache.flink.streaming.api.CheckpointingMode mode)

getCheckpointTimeoutMillis

@Default.Long(value=1200000L)
java.lang.Long getCheckpointTimeoutMillis()

setCheckpointTimeoutMillis

void setCheckpointTimeoutMillis(java.lang.Long checkpointTimeoutMillis)

getNumberOfExecutionRetries

@Default.Integer(value=-1)
java.lang.Integer getNumberOfExecutionRetries()

setNumberOfExecutionRetries

void setNumberOfExecutionRetries(java.lang.Integer retries)

getExecutionRetryDelay

@Default.Long(value=-1L)
java.lang.Long getExecutionRetryDelay()

setExecutionRetryDelay

void setExecutionRetryDelay(java.lang.Long delay)

getObjectReuse

@Default.Boolean(value=false)
java.lang.Boolean getObjectReuse()

setObjectReuse

void setObjectReuse(java.lang.Boolean reuse)

getStateBackend

org.apache.flink.runtime.state.AbstractStateBackend getStateBackend()

State backend to store Beam's state during computation. Note: Only applicable when executing in streaming mode.

setStateBackend

void setStateBackend(org.apache.flink.runtime.state.AbstractStateBackend stateBackend)

getEnableMetrics

@Default.Boolean(value=true)
java.lang.Boolean getEnableMetrics()

setEnableMetrics

void setEnableMetrics(java.lang.Boolean enableMetrics)

isExternalizedCheckpointsEnabled

@Default.Boolean(value=false)
java.lang.Boolean isExternalizedCheckpointsEnabled()

Enables or disables externalized checkpoints.

setExternalizedCheckpointsEnabled

void setExternalizedCheckpointsEnabled(java.lang.Boolean externalCheckpoints)

getRetainExternalizedCheckpointsOnCancellation

@Default.Boolean(value=false)
java.lang.Boolean getRetainExternalizedCheckpointsOnCancellation()

setRetainExternalizedCheckpointsOnCancellation

void setRetainExternalizedCheckpointsOnCancellation(java.lang.Boolean retainOnCancellation)

getMaxBundleSize

@Default.Long(value=1000L)
java.lang.Long getMaxBundleSize()

setMaxBundleSize

void setMaxBundleSize(java.lang.Long size)

getMaxBundleTimeMills

@Default.Long(value=1000L)
java.lang.Long getMaxBundleTimeMills()

setMaxBundleTimeMills

void setMaxBundleTimeMills(java.lang.Long time)

isShutdownSourcesOnFinalWatermark

@Default.Boolean(value=false)
java.lang.Boolean isShutdownSourcesOnFinalWatermark()

Whether to shutdown sources when their watermark reaches +Inf. For production use cases you want this to be disabled because Flink will currently (versions <= 1.5) stop doing checkpoints when any operator (which includes sources) is finished.

Please see FLINK-2491 for progress on this issue.

setShutdownSourcesOnFinalWatermark

void setShutdownSourcesOnFinalWatermark(java.lang.Boolean shutdownOnFinalWatermark)