org.apache.beam.sdk.io.hadoop.format

Class HDFSSynchronization

java.lang.Object

org.apache.beam.sdk.io.hadoop.format.HDFSSynchronization

All Implemented Interfaces:

java.io.Serializable, ExternalSynchronization

public class HDFSSynchronization
extends java.lang.Object
implements ExternalSynchronization

Implementation of ExternalSynchronization which registers locks in the HDFS.

Requires locksDir to be specified. This directory MUST be different that directory which is possibly stored under "mapreduce.output.fileoutputformat.outputdir" key. Otherwise setup of job will fail because the directory will exist before job setup.

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static interface	HDFSSynchronization.ThrowingFunction<T1,T2,X extends java.lang.Exception> Function which can throw exception.

Constructor Summary

Constructors

Constructor and Description
HDFSSynchronization(java.lang.String locksDir) Creates instance of HDFSSynchronization.

Method Summary

Modifier and Type	Method and Description
org.apache.hadoop.mapreduce.TaskAttemptID	acquireTaskAttemptIdLock(org.apache.hadoop.conf.Configuration conf, int taskId) Creates unique TaskAttemptID for given taskId.
org.apache.hadoop.mapreduce.TaskID	acquireTaskIdLock(org.apache.hadoop.conf.Configuration conf) Creates TaskID with unique id among given job.
void	releaseJobIdLock(org.apache.hadoop.conf.Configuration conf) Deletes lock ids bounded with given job if any exists.
boolean	tryAcquireJobLock(org.apache.hadoop.conf.Configuration conf) Tries to acquire lock for given job.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

HDFSSynchronization

public HDFSSynchronization(java.lang.String locksDir)

Creates instance of HDFSSynchronization.

Parameters:

locksDir - directory where locks will be stored. This directory MUST be different that directory which is possibly stored under "mapreduce.output.fileoutputformat.outputdir" key. Otherwise setup of job will fail because the directory will exist before job setup.

Method Detail

tryAcquireJobLock

public boolean tryAcquireJobLock(org.apache.hadoop.conf.Configuration conf)

Description copied from interface: ExternalSynchronization

Tries to acquire lock for given job.

Specified by:

tryAcquireJobLock in interface ExternalSynchronization

Parameters:

conf - configuration bounded with given job.

Returns:

true if the lock was acquired, false otherwise.

releaseJobIdLock

public void releaseJobIdLock(org.apache.hadoop.conf.Configuration conf)

Description copied from interface: ExternalSynchronization

Deletes lock ids bounded with given job if any exists.

Specified by:

releaseJobIdLock in interface ExternalSynchronization

Parameters:

conf - hadoop configuration of given job.

acquireTaskIdLock

public org.apache.hadoop.mapreduce.TaskID acquireTaskIdLock(org.apache.hadoop.conf.Configuration conf)

Description copied from interface: ExternalSynchronization

Creates TaskID with unique id among given job.

Specified by:

acquireTaskIdLock in interface ExternalSynchronization

Parameters:

conf - hadoop configuration of given job.

Returns:

TaskID with unique id among given job.

acquireTaskAttemptIdLock

public org.apache.hadoop.mapreduce.TaskAttemptID acquireTaskAttemptIdLock(org.apache.hadoop.conf.Configuration conf,
                                                                          int taskId)

Description copied from interface: ExternalSynchronization

Creates unique TaskAttemptID for given taskId.

Specified by:

acquireTaskAttemptIdLock in interface ExternalSynchronization

Parameters:

conf - configuration of given task and job

taskId - id of the task

Returns:

Unique TaskAttemptID for given taskId.