org.apache.beam.runners.apex

Class ApexYarnLauncher

java.lang.Object

org.apache.beam.runners.apex.ApexYarnLauncher

public class ApexYarnLauncher
extends java.lang.Object

Proxy to launch the YARN application through the hadoop script to run in the pre-configured environment (class path, configuration, native libraries etc.).

The proxy takes the DAG and communicates with the Hadoop services to launch it on the cluster.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
protected static class	ApexYarnLauncher.LaunchParams Launch parameters that will be serialized and passed to the child process.
static class	ApexYarnLauncher.ProcessWatcher Starts a command and waits for it to complete.

Constructor Summary

Constructors

Constructor and Description
ApexYarnLauncher()

Method Summary

Modifier and Type	Method and Description
static void	addProperties(org.apache.hadoop.conf.Configuration conf, java.util.Properties props) Transfer the properties to the configuration object.
static void	createJar(java.io.File dir, java.io.File jarFile) Create a jar file from the given directory.
static java.util.List<java.io.File>	getYarnDeployDependencies() From the current classpath, find the jar files that need to be deployed with the application to run on YARN.
protected org.apache.apex.api.Launcher.AppHandle	launchApp(ApexYarnLauncher.LaunchParams params)
org.apache.apex.api.Launcher.AppHandle	launchApp(com.datatorrent.api.StreamingApplication app, java.util.Properties configProperties)
static void	main(java.lang.String[] args) The main method expects the serialized DAG and will launch the YARN application.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

ApexYarnLauncher

public ApexYarnLauncher()

Method Detail

launchApp

public org.apache.apex.api.Launcher.AppHandle launchApp(com.datatorrent.api.StreamingApplication app,
                                                        java.util.Properties configProperties)
                                                 throws java.io.IOException

Throws:

java.io.IOException

launchApp

protected org.apache.apex.api.Launcher.AppHandle launchApp(ApexYarnLauncher.LaunchParams params)
                                                    throws java.io.IOException

Throws:

java.io.IOException

getYarnDeployDependencies

public static java.util.List<java.io.File> getYarnDeployDependencies()
                                                              throws java.io.IOException

From the current classpath, find the jar files that need to be deployed with the application to run on YARN. Hadoop dependencies are provided through the Hadoop installation and the application should not bundle them to avoid conflicts. This is done by removing the Hadoop compile dependencies (transitively) by parsing the Maven dependency tree.

Returns:

list of jar files to ship

Throws:

java.io.IOException - when dependency information cannot be read

createJar

public static void createJar(java.io.File dir,
                             java.io.File jarFile)
                      throws java.io.IOException

Create a jar file from the given directory.

Parameters:

dir - source directory

jarFile - jar file name

Throws:

java.io.IOException - when file cannot be created

addProperties

public static void addProperties(org.apache.hadoop.conf.Configuration conf,
                                 java.util.Properties props)

Transfer the properties to the configuration object.

main

public static void main(java.lang.String[] args)
                 throws java.io.IOException

The main method expects the serialized DAG and will launch the YARN application.

Parameters:

args - location of launch parameters

Throws:

java.io.IOException - when parameters cannot be read