org.apache.beam.sdk.managed

Class Managed

java.lang.Object

org.apache.beam.sdk.managed.Managed

public class Managed
extends java.lang.Object

Top-level PTransforms that build and instantiate turnkey transforms.

Available transforms

This API currently supports two operations: read(java.lang.String) and write(java.lang.String). Each one enumerates the available transforms in a TRANSFORMS map.

Building a Managed turnkey transform

Turnkey transforms are represented as SchemaTransforms, which means each one has a defined configuration. A given transform can be built with a Map<String, Object> that specifies arguments using like so:

 PCollectionRowTuple output = PCollectionRowTuple.empty(pipeline).apply(
       Managed.read(ICEBERG)
           .withConfig(ImmutableMap.<String, Object>.builder()
               .put("foo", "abc")
               .put("bar", 123)
               .build()));
 

Instead of specifying configuration arguments directly in the code, one can provide the location to a YAML file that contains this information. Say we have the following YAML file:

 foo: "abc"
 bar: 123
 

The file's path can be passed in to the Managed API like so:

 PCollectionRowTuple input = PCollectionRowTuple.of("input", pipeline.apply(Create.of(...)))

 PCollectionRowTuple output = input.apply(
     Managed.write(ICEBERG)
         .withConfigUrl(<config path>));
 

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	Managed.ManagedTransform

Field Summary

Fields

Modifier and Type	Field and Description
static java.lang.String	ICEBERG
static java.util.Map<java.lang.String,java.lang.String>	READ_TRANSFORMS
static java.util.Map<java.lang.String,java.lang.String>	WRITE_TRANSFORMS

Constructor Summary

Constructors

Constructor and Description
Managed()

Method Summary

Modifier and Type	Method and Description
static Managed.ManagedTransform	read(java.lang.String source) Instantiates a Managed.ManagedTransform transform for the specified source.
static Managed.ManagedTransform	write(java.lang.String sink) Instantiates a Managed.ManagedTransform transform for the specified sink.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

ICEBERG

public static final java.lang.String ICEBERG

See Also:

Constant Field Values

READ_TRANSFORMS

public static final java.util.Map<java.lang.String,java.lang.String> READ_TRANSFORMS

WRITE_TRANSFORMS

public static final java.util.Map<java.lang.String,java.lang.String> WRITE_TRANSFORMS

Constructor Detail

Managed

public Managed()

Method Detail

read

public static Managed.ManagedTransform read(java.lang.String source)

Instantiates a Managed.ManagedTransform transform for the specified source. The supported managed sources are:

• ICEBERG : Read from Apache Iceberg

write

public static Managed.ManagedTransform write(java.lang.String sink)

Instantiates a Managed.ManagedTransform transform for the specified sink. The supported managed sinks are:

• ICEBERG : Write to Apache Iceberg