org.apache.beam.sdk.options

Interface SdkHarnessOptions

All Superinterfaces:

HasDisplayData, PipelineOptions

public interface SdkHarnessOptions
extends PipelineOptions

Options that are used to control configuration of the SDK harness.

Nested Class Summary

Nested Classes

Modifier and Type	Interface and Description
static class	SdkHarnessOptions.DefaultMaxCacheMemoryUsageMb The default implementation which detects how much memory to use for a process wide cache.
static class	SdkHarnessOptions.DefaultMaxCacheMemoryUsageMbFactory A DefaultValueFactory which constructs an instance of the class specified by maxCacheMemoryUsageMbClass to compute the maximum amount of memory to allocate to the process wide cache within an SDK harness instance.
static class	SdkHarnessOptions.LogLevel The set of log levels that can be used in the SDK harness.
static interface	SdkHarnessOptions.MaxCacheMemoryUsageMb Specifies the maximum amount of memory to use within the current SDK harness instance.
static class	SdkHarnessOptions.SdkHarnessLogLevelOverrides Defines a log level override for a specific class, package, or name.

Nested classes/interfaces inherited from interface org.apache.beam.sdk.options.PipelineOptions

PipelineOptions.AtomicLongFactory, PipelineOptions.CheckEnabled, PipelineOptions.DirectRunner, PipelineOptions.JobNameFactory, PipelineOptions.UserAgentFactory

Method Summary

Modifier and Type	Method and Description
static java.util.List<java.util.logging.Logger>	getConfiguredLoggerFromOptions(SdkHarnessOptions loggingOptions) Configure log manager's default log level and log level overrides from the sdk harness options, and return the list of configured loggers.
SdkHarnessOptions.LogLevel	getDefaultSdkHarnessLogLevel() This option controls the default log level of all loggers without a log level override.
int	getGroupingTableMaxSizeMb() Size (in MB) of each grouping table used to pre-combine elements.
java.util.List<java.lang.String>	getJdkAddOpenModules() Open modules needed for reflection that access JDK internals with Java 9+
@NonNegative int	getMaxCacheMemoryUsageMb() Size (in MB) for the process wide cache within the SDK harness.
java.lang.Class<? extends SdkHarnessOptions.MaxCacheMemoryUsageMb>	getMaxCacheMemoryUsageMbClass() An instance of this class will be used to specify the maximum amount of memory to allocate to a cache within an SDK harness instance.
@NonNegative float	getMaxCacheMemoryUsagePercent() Size (in % [0 - 100]) for the process wide cache within the SDK harness.
SdkHarnessOptions.SdkHarnessLogLevelOverrides	getSdkHarnessLogLevelOverrides() This option controls the log levels for specifically named loggers.
void	setDefaultSdkHarnessLogLevel(SdkHarnessOptions.LogLevel logLevel)
void	setGroupingTableMaxSizeMb(int value)
void	setJdkAddOpenModules(java.util.List<java.lang.String> options)
void	setMaxCacheMemoryUsageMb(@NonNegative int value)
void	setMaxCacheMemoryUsageMbClass(java.lang.Class<? extends SdkHarnessOptions.MaxCacheMemoryUsageMb> kls)
void	setMaxCacheMemoryUsagePercent(@NonNegative float value)
void	setSdkHarnessLogLevelOverrides(SdkHarnessOptions.SdkHarnessLogLevelOverrides value)

Methods inherited from interface org.apache.beam.sdk.options.PipelineOptions

as, getJobName, getOptionsId, getRunner, getStableUniqueNames, getTempLocation, getUserAgent, outputRuntimeOptions, setJobName, setOptionsId, setRunner, setStableUniqueNames, setTempLocation, setUserAgent

Methods inherited from interface org.apache.beam.sdk.transforms.display.HasDisplayData

populateDisplayData

Method Detail

getDefaultSdkHarnessLogLevel

@Default.Enum(value="INFO")
SdkHarnessOptions.LogLevel getDefaultSdkHarnessLogLevel()

This option controls the default log level of all loggers without a log level override.

setDefaultSdkHarnessLogLevel

void setDefaultSdkHarnessLogLevel(SdkHarnessOptions.LogLevel logLevel)

getSdkHarnessLogLevelOverrides

SdkHarnessOptions.SdkHarnessLogLevelOverrides getSdkHarnessLogLevelOverrides()

This option controls the log levels for specifically named loggers.

Later options with equivalent names override earlier options.

See SdkHarnessOptions.SdkHarnessLogLevelOverrides for more information on how to configure logging on a per Class, Package, or name basis. If used from the command line, the expected format is {"Name":"LogLevel",...}, further details on SdkHarnessOptions.SdkHarnessLogLevelOverrides.from(java.util.Map<java.lang.String, java.lang.String>).

setSdkHarnessLogLevelOverrides

void setSdkHarnessLogLevelOverrides(SdkHarnessOptions.SdkHarnessLogLevelOverrides value)

getGroupingTableMaxSizeMb

@Default.Integer(value=100)
int getGroupingTableMaxSizeMb()

Size (in MB) of each grouping table used to pre-combine elements. Larger values may reduce the amount of data shuffled. If unset, defaults to 100 MB.

CAUTION: If set too large, workers may run into OOM conditions more easily, each worker may have many grouping tables in-memory concurrently.

CAUTION: This option does not apply to portable runners such as Dataflow Prime. See setMaxCacheMemoryUsageMb(int), setMaxCacheMemoryUsagePercent(float), or setMaxCacheMemoryUsageMbClass(java.lang.Class<? extends org.apache.beam.sdk.options.SdkHarnessOptions.MaxCacheMemoryUsageMb>) to configure memory thresholds that apply to the grouping table and other cached objects.

setGroupingTableMaxSizeMb

void setGroupingTableMaxSizeMb(int value)

getMaxCacheMemoryUsageMb

@Default.InstanceFactory(value=SdkHarnessOptions.DefaultMaxCacheMemoryUsageMbFactory.class)
@NonNegative int getMaxCacheMemoryUsageMb()

Size (in MB) for the process wide cache within the SDK harness. The cache is responsible for storing all values which are cached within a bundle and across bundles such as side inputs and user state.

CAUTION: If set too large, SDK harness instances may run into OOM conditions more easily.

See SdkHarnessOptions.DefaultMaxCacheMemoryUsageMbFactory for details on how maxCacheMemoryUsageMb is computed if this parameter is unspecified.

setMaxCacheMemoryUsageMb

void setMaxCacheMemoryUsageMb(@NonNegative int value)

getMaxCacheMemoryUsagePercent

@Default.Float(value=20.0f)
@NonNegative float getMaxCacheMemoryUsagePercent()

Size (in % [0 - 100]) for the process wide cache within the SDK harness. The cache is responsible for storing all values which are cached within a bundle and across bundles such as side inputs and user state.

This parameter will only be used if an explicit value was not specified for maxCacheMemoryUsageMb.

setMaxCacheMemoryUsagePercent

void setMaxCacheMemoryUsagePercent(@NonNegative float value)

getMaxCacheMemoryUsageMbClass

@Default.Class(value=SdkHarnessOptions.DefaultMaxCacheMemoryUsageMb.class)
java.lang.Class<? extends SdkHarnessOptions.MaxCacheMemoryUsageMb> getMaxCacheMemoryUsageMbClass()

An instance of this class will be used to specify the maximum amount of memory to allocate to a cache within an SDK harness instance.

This parameter will only be used if an explicit value was not specified for maxCacheMemoryUsageMb.

See SdkHarnessOptions.DefaultMaxCacheMemoryUsageMb for details on how maxCacheMemoryUsageMb is computed if this parameter is unspecified.

setMaxCacheMemoryUsageMbClass

void setMaxCacheMemoryUsageMbClass(java.lang.Class<? extends SdkHarnessOptions.MaxCacheMemoryUsageMb> kls)

getJdkAddOpenModules

java.util.List<java.lang.String> getJdkAddOpenModules()

Open modules needed for reflection that access JDK internals with Java 9+

With JDK 16+, JDK internals are strongly encapsulated and can result in an InaccessibleObjectException being thrown if a tool or library uses reflection that access JDK internals. If you see these errors in your worker logs, you can pass in modules to open using the format module/package=target-module(,target-module)* to allow access to the library. E.g. java.base/java.lang=jamm

You may see warnings that jamm, a library used to more accurately size objects, is unable to make a private field accessible. To resolve the warning, open the specified module/package to jamm.

setJdkAddOpenModules

void setJdkAddOpenModules(java.util.List<java.lang.String> options)

getConfiguredLoggerFromOptions

static java.util.List<java.util.logging.Logger> getConfiguredLoggerFromOptions(SdkHarnessOptions loggingOptions)

Configure log manager's default log level and log level overrides from the sdk harness options, and return the list of configured loggers.