Package org.apache.beam.sdk.options

Interface SdkHarnessOptions

All Superinterfaces:
HasDisplayData, MemoryMonitorOptions, PipelineOptions
public interface SdkHarnessOptions extends PipelineOptions, MemoryMonitorOptions
Options that are used to control configuration of the SDK harness.

  • Method Details

    • getDefaultSdkHarnessLogLevel

      @Enum("INFO") SdkHarnessOptions.LogLevel getDefaultSdkHarnessLogLevel()
      This option controls the default log level of all loggers without a log level override.

    • setDefaultSdkHarnessLogLevel

      void setDefaultSdkHarnessLogLevel(SdkHarnessOptions.LogLevel logLevel)

    • getSdkHarnessLogLevelOverrides

      SdkHarnessOptions.SdkHarnessLogLevelOverrides getSdkHarnessLogLevelOverrides()
      This option controls the log levels for specifically named loggers.

      Later options with equivalent names override earlier options.

      See SdkHarnessOptions.SdkHarnessLogLevelOverrides for more information on how to configure logging on a per Class, Package, or name basis. If used from the command line, the expected format is {"Name":"LogLevel",...}, further details on SdkHarnessOptions.SdkHarnessLogLevelOverrides.from(java.util.Map<java.lang.String, java.lang.String>).

    • setSdkHarnessLogLevelOverrides

      void setSdkHarnessLogLevelOverrides(SdkHarnessOptions.SdkHarnessLogLevelOverrides value)

    • getLogMdc

      @Boolean(true) boolean getLogMdc()
      Whether to include SLF4J MDC in log entries.

    • setLogMdc

      void setLogMdc(boolean value)

    • getEnableLogViaFnApi

      @Boolean(true) @Hidden boolean getEnableLogViaFnApi()
      This option controls whether logging will be redirected through the FnApi.

    • setEnableLogViaFnApi

      void setEnableLogViaFnApi(boolean enableLogViaFnApi)

    • getGroupingTableMaxSizeMb

      @Integer(100) int getGroupingTableMaxSizeMb()
      Size (in MB) of each grouping table used to pre-combine elements. Larger values may reduce the amount of data shuffled. If unset, defaults to 100 MB.

      CAUTION: If set too large, workers may run into OOM conditions more easily, each worker may have many grouping tables in-memory concurrently.

      CAUTION: This option does not apply to portable runners such as Dataflow Prime. See setMaxCacheMemoryUsageMb(int), setMaxCacheMemoryUsagePercent(float), or setMaxCacheMemoryUsageMbClass(java.lang.Class<? extends org.apache.beam.sdk.options.SdkHarnessOptions.MaxCacheMemoryUsageMb>) to configure memory thresholds that apply to the grouping table and other cached objects.

    • setGroupingTableMaxSizeMb

      void setGroupingTableMaxSizeMb(int value)

    • getMaxCacheMemoryUsageMb

      @InstanceFactory(DefaultMaxCacheMemoryUsageMbFactory.class) @org.checkerframework.checker.index.qual.NonNegative int getMaxCacheMemoryUsageMb()
      Size (in MB) for the process wide cache within the SDK harness. The cache is responsible for storing all values which are cached within a bundle and across bundles such as side inputs and user state.

      CAUTION: If set too large, SDK harness instances may run into OOM conditions more easily.

      See SdkHarnessOptions.DefaultMaxCacheMemoryUsageMbFactory for details on how maxCacheMemoryUsageMb is computed if this parameter is unspecified.

    • setMaxCacheMemoryUsageMb

      void setMaxCacheMemoryUsageMb(@org.checkerframework.checker.index.qual.NonNegative int value)

    • getMaxCacheMemoryUsagePercent

      @Float(20.0f) @org.checkerframework.checker.index.qual.NonNegative float getMaxCacheMemoryUsagePercent()
      Size (in % [0 - 100]) for the process wide cache within the SDK harness. The cache is responsible for storing all values which are cached within a bundle and across bundles such as side inputs and user state.

      This parameter will only be used if an explicit value was not specified for maxCacheMemoryUsageMb.

    • setMaxCacheMemoryUsagePercent

      void setMaxCacheMemoryUsagePercent(@org.checkerframework.checker.index.qual.NonNegative float value)

    • getMaxCacheMemoryUsageMbClass

      @Class(DefaultMaxCacheMemoryUsageMb.class) Class<? extends SdkHarnessOptions.MaxCacheMemoryUsageMb> getMaxCacheMemoryUsageMbClass()
      An instance of this class will be used to specify the maximum amount of memory to allocate to a cache within an SDK harness instance.

      This parameter will only be used if an explicit value was not specified for maxCacheMemoryUsageMb.

      See SdkHarnessOptions.DefaultMaxCacheMemoryUsageMb for details on how maxCacheMemoryUsageMb is computed if this parameter is unspecified.

    • setMaxCacheMemoryUsageMbClass

      void setMaxCacheMemoryUsageMbClass(Class<? extends SdkHarnessOptions.MaxCacheMemoryUsageMb> kls)

    • getJdkAddOpenModules

      List<String> getJdkAddOpenModules()
      Open modules needed for reflection that access JDK internals with Java 9+.

      With JDK 16+, JDK internals are strongly encapsulated and can result in an InaccessibleObjectException being thrown if a tool or library uses reflection that access JDK internals. If you see these errors in your worker logs, you can pass in modules to open using the format module/package=target-module[,module2/package2=another-target-module] to allow access to the library. E.g. --jdkAddOpenModules=java.base/java.lang=jamm. This will set --add-opens JVM flag in SDK Harness invocation.

      You may see warnings that jamm, a library used to more accurately size objects, is unable to make a private field accessible. To resolve the warning, open the specified module/package to jamm.

    • setJdkAddOpenModules

      void setJdkAddOpenModules(List<String> options)

    • getJdkAddRootModules

      List<String> getJdkAddRootModules()
      Add modules to the default root set with Java 11+.

      Set --add-modules JVM flag in SDK Harness invocation. E.g. --jdkAddModules=module1,module2.

    • setJdkAddRootModules

      void setJdkAddRootModules(List<String> options)

    • getConfiguredLoggerFromOptions

      static List<Logger> getConfiguredLoggerFromOptions(SdkHarnessOptions loggingOptions)
      Configure log manager's default log level and log level overrides from the sdk harness options, and return the list of configured loggers.

    • getBundleProcessorCacheTimeout

      @Hidden @InstanceFactory(BundleProcessorCacheTimeoutFactory.class) Duration getBundleProcessorCacheTimeout()

    • setBundleProcessorCacheTimeout

      void setBundleProcessorCacheTimeout(Duration duration)

    • getElementProcessingTimeoutMinutes

      @org.checkerframework.checker.index.qual.NonNegative int getElementProcessingTimeoutMinutes()
      The time limit (in minute) that an SDK worker allows for a PTransform operation before signaling the runner harness to restart the SDK worker.

    • setElementProcessingTimeoutMinutes

      void setElementProcessingTimeoutMinutes(int value)