org.apache.beam.sdk.io.cdap

Class Plugin<K,V>

java.lang.Object

org.apache.beam.sdk.io.cdap.Plugin<K,V>

public abstract class Plugin<K,V>
extends java.lang.Object

Class wrapper for a CDAP plugin.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	Plugin.Builder<K,V> Builder class for a Plugin.

Field Summary

Fields

Modifier and Type	Field and Description
protected SubmitterLifecycle	cdapPluginObj
protected org.apache.hadoop.conf.Configuration	hadoopConfiguration
protected PluginConfig	pluginConfig

Constructor Summary

Constructors

Constructor and Description
Plugin()

Method Summary

Modifier and Type	Method and Description
static <K,V> Plugin.Builder<K,V>	builder() Creates a plugin builder instance.
static <K,V> Plugin<K,V>	createBatch(java.lang.Class<?> newPluginClass, java.lang.Class<?> newFormatClass, java.lang.Class<?> newFormatProviderClass) Creates a batch plugin instance.
static <K,V> Plugin<K,V>	createStreaming(java.lang.Class<?> newPluginClass, SerializableFunction<V,java.lang.Long> getOffsetFn, java.lang.Class<? extends org.apache.spark.streaming.receiver.Receiver<V>> receiverClass) Creates a streaming plugin instance with default function for getting args for Receiver.
static <K,V> Plugin<K,V>	createStreaming(java.lang.Class<?> newPluginClass, SerializableFunction<V,java.lang.Long> getOffsetFn, java.lang.Class<? extends org.apache.spark.streaming.receiver.Receiver<V>> receiverClass, SerializableFunction<PluginConfig,java.lang.Object[]> getReceiverArgsFromConfigFn) Creates a streaming plugin instance.
abstract BatchContextImpl	getContext() Gets the context of a plugin.
abstract java.lang.Class<?>	getFormatClass() Gets InputFormat or OutputFormat class for a plugin.
abstract java.lang.Class<?>	getFormatProviderClass() Gets InputFormatProvider or OutputFormatProvider class for a plugin.
abstract SerializableFunction<V,java.lang.Long>	getGetOffsetFn() Gets a SerializableFunction that defines how to get record offset for CDAP Plugin class.
abstract SerializableFunction<PluginConfig,java.lang.Object[]>	getGetReceiverArgsFromConfigFn() Gets a SerializableFunction that defines how to get constructor arguments for Receiver using PluginConfig.
org.apache.hadoop.conf.Configuration	getHadoopConfiguration() Gets a plugin Hadoop configuration.
abstract java.lang.Class<?>	getPluginClass() Gets the main class of a plugin.
PluginConfig	getPluginConfig() Gets a plugin config.
abstract PluginConstants.PluginType	getPluginType() Gets a plugin type.
ReceiverBuilder<V,? extends org.apache.spark.streaming.receiver.Receiver<V>>	getReceiverBuilder() Gets a ReceiverBuilder.
abstract java.lang.Class<? extends org.apache.spark.streaming.receiver.Receiver<V>>	getReceiverClass() Gets Spark Receiver class for a CDAP plugin.
static BatchContextImpl	initContext(java.lang.Class<?> cdapPluginClass) Initializes BatchContextImpl for CDAP plugin.
static PluginConstants.PluginType	initPluginType(java.lang.Class<?> pluginClass) Gets value of a plugin type.
java.lang.Boolean	isUnbounded() Gets value of a plugin type.
void	prepareRun() Calls SubmitterLifecycle.prepareRun(Object) method on the cdapPluginObj passing needed configuration object as a parameter.
Plugin<K,V>	withConfig(PluginConfig pluginConfig) Sets a plugin config.
Plugin<K,V>	withHadoopConfiguration(java.lang.Class<K> formatKeyClass, java.lang.Class<V> formatValueClass) Sets a plugin Hadoop configuration.
Plugin<K,V>	withHadoopConfiguration(org.apache.hadoop.conf.Configuration hadoopConfiguration) Sets a plugin Hadoop configuration.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

pluginConfig

@Nullable
protected PluginConfig pluginConfig

hadoopConfiguration

@Nullable
protected org.apache.hadoop.conf.Configuration hadoopConfiguration

cdapPluginObj

@Nullable
protected SubmitterLifecycle cdapPluginObj

Constructor Detail

Plugin

public Plugin()

Method Detail

getContext

public abstract BatchContextImpl getContext()

Gets the context of a plugin.

getPluginClass

public abstract java.lang.Class<?> getPluginClass()

Gets the main class of a plugin.

getFormatClass

@Nullable
public abstract java.lang.Class<?> getFormatClass()

Gets InputFormat or OutputFormat class for a plugin.

getFormatProviderClass

@Nullable
public abstract java.lang.Class<?> getFormatProviderClass()

Gets InputFormatProvider or OutputFormatProvider class for a plugin.

getReceiverClass

@Nullable
public abstract java.lang.Class<? extends org.apache.spark.streaming.receiver.Receiver<V>> getReceiverClass()

Gets Spark Receiver class for a CDAP plugin.

getGetOffsetFn

@Nullable
public abstract SerializableFunction<V,java.lang.Long> getGetOffsetFn()

Gets a SerializableFunction that defines how to get record offset for CDAP Plugin class.

getGetReceiverArgsFromConfigFn

@Nullable
public abstract SerializableFunction<PluginConfig,java.lang.Object[]> getGetReceiverArgsFromConfigFn()

Gets a SerializableFunction that defines how to get constructor arguments for Receiver using PluginConfig.

withConfig

public Plugin<K,V> withConfig(PluginConfig pluginConfig)

Sets a plugin config.

getPluginConfig

@Nullable
public PluginConfig getPluginConfig()

Gets a plugin config.

prepareRun

public void prepareRun()

Calls SubmitterLifecycle.prepareRun(Object) method on the cdapPluginObj passing needed configuration object as a parameter. This method is needed for validating connection to the CDAP sink/source and performing initial tuning.

withHadoopConfiguration

public Plugin<K,V> withHadoopConfiguration(java.lang.Class<K> formatKeyClass,
                                           java.lang.Class<V> formatValueClass)

Sets a plugin Hadoop configuration.

withHadoopConfiguration

public Plugin<K,V> withHadoopConfiguration(org.apache.hadoop.conf.Configuration hadoopConfiguration)

Sets a plugin Hadoop configuration.

getHadoopConfiguration

public org.apache.hadoop.conf.Configuration getHadoopConfiguration()

Gets a plugin Hadoop configuration.

getPluginType

public abstract PluginConstants.PluginType getPluginType()

Gets a plugin type.

initPluginType

public static PluginConstants.PluginType initPluginType(java.lang.Class<?> pluginClass)
                                                 throws java.lang.IllegalArgumentException

Gets value of a plugin type.

Throws:

java.lang.IllegalArgumentException

initContext

public static BatchContextImpl initContext(java.lang.Class<?> cdapPluginClass)

Initializes BatchContextImpl for CDAP plugin.

isUnbounded

public java.lang.Boolean isUnbounded()

Gets value of a plugin type.

getReceiverBuilder

public ReceiverBuilder<V,? extends org.apache.spark.streaming.receiver.Receiver<V>> getReceiverBuilder()

Gets a ReceiverBuilder.

createBatch

public static <K,V> Plugin<K,V> createBatch(java.lang.Class<?> newPluginClass,
                                            java.lang.Class<?> newFormatClass,
                                            java.lang.Class<?> newFormatProviderClass)

Creates a batch plugin instance.

Parameters:

newPluginClass - class of the CDAP plugin Plugin.

newFormatClass - Hadoop Input or Output format class.

newFormatProviderClass - Hadoop Input or Output format provider class.

createStreaming

public static <K,V> Plugin<K,V> createStreaming(java.lang.Class<?> newPluginClass,
                                                SerializableFunction<V,java.lang.Long> getOffsetFn,
                                                java.lang.Class<? extends org.apache.spark.streaming.receiver.Receiver<V>> receiverClass,
                                                SerializableFunction<PluginConfig,java.lang.Object[]> getReceiverArgsFromConfigFn)

Creates a streaming plugin instance.

Parameters:

newPluginClass - class of the CDAP plugin Plugin.

getOffsetFn - SerializableFunction that defines how to get record offset for CDAP Plugin class.

receiverClass - Spark Receiver class for a CDAP plugin.

getReceiverArgsFromConfigFn - SerializableFunction that defines how to get constructor arguments for Receiver using PluginConfig.

createStreaming

public static <K,V> Plugin<K,V> createStreaming(java.lang.Class<?> newPluginClass,
                                                SerializableFunction<V,java.lang.Long> getOffsetFn,
                                                java.lang.Class<? extends org.apache.spark.streaming.receiver.Receiver<V>> receiverClass)

Creates a streaming plugin instance with default function for getting args for Receiver.

Parameters:

newPluginClass - class of the CDAP plugin Plugin.

getOffsetFn - SerializableFunction that defines how to get record offset for CDAP Plugin class.

receiverClass - Spark Receiver class for a CDAP plugin.

builder

public static <K,V> Plugin.Builder<K,V> builder()

Creates a plugin builder instance.