org.apache.beam.sdk.managed

Class Managed

java.lang.Object

org.apache.beam.sdk.managed.Managed

public class Managed
extends java.lang.Object

Top-level PTransforms that build and instantiate turnkey transforms.

Available transforms

This API currently supports two operations: read(java.lang.String) and write(java.lang.String). Each one enumerates the available transforms in a TRANSFORMS map.

Building a Managed turnkey transform

Turnkey transforms are represented as SchemaTransforms, which means each one has a defined configuration. A given transform can be built with a Map<String, Object> that specifies arguments using like so:

 PCollection<Row> rows = pipeline.apply(
       Managed.read(ICEBERG)
           .withConfig(ImmutableMap.<String, Object>.builder()
               .put("foo", "abc")
               .put("bar", 123)
               .build()))
       .getOutput();
 

Instead of specifying configuration arguments directly in the code, one can provide the location to a YAML file that contains this information. Say we have the following config.yaml file:

 foo: "abc"
 bar: 123
 

The file's path can be passed in to the Managed API like so:

 PCollection<Row> inputRows = pipeline.apply(Create.of(...));

 inputRows.apply(Managed.write(ICEBERG).withConfigUrl("path/to/config.yaml"));
 

Runner specific features

Google Cloud Dataflow supports additional management features for Managed including automatically upgrading transforms to the latest supported version. For more details and examples, please see Dataflow managed I/O.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	Managed.ManagedTransform

Field Summary

Fields

Modifier and Type	Field and Description
static java.lang.String	BIGQUERY
static java.lang.String	ICEBERG
static java.lang.String	ICEBERG_CDC
static java.lang.String	KAFKA
static java.util.Map<java.lang.String,java.lang.String>	READ_TRANSFORMS
static java.util.Map<java.lang.String,java.lang.String>	WRITE_TRANSFORMS

Constructor Summary

Constructors

Constructor and Description
Managed()

Method Summary

Modifier and Type	Method and Description
static Managed.ManagedTransform	read(java.lang.String source) Instantiates a Managed.ManagedTransform transform for the specified source.
static Managed.ManagedTransform	write(java.lang.String sink) Instantiates a Managed.ManagedTransform transform for the specified sink.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

ICEBERG

public static final java.lang.String ICEBERG

See Also:

Constant Field Values

ICEBERG_CDC

public static final java.lang.String ICEBERG_CDC

See Also:

Constant Field Values

KAFKA

public static final java.lang.String KAFKA

See Also:

Constant Field Values

BIGQUERY

public static final java.lang.String BIGQUERY

See Also:

Constant Field Values

READ_TRANSFORMS

public static final java.util.Map<java.lang.String,java.lang.String> READ_TRANSFORMS

WRITE_TRANSFORMS

public static final java.util.Map<java.lang.String,java.lang.String> WRITE_TRANSFORMS

Constructor Detail

Managed

public Managed()

Method Detail

read

public static Managed.ManagedTransform read(java.lang.String source)

Instantiates a Managed.ManagedTransform transform for the specified source. The supported managed sources are:

• ICEBERG : Read from Apache Iceberg tables using IcebergIO
• ICEBERG_CDC : CDC Read from Apache Iceberg tables using IcebergIO
• KAFKA : Read from Apache Kafka topics using KafkaIO
• BIGQUERY : Read from GCP BigQuery tables using BigQueryIO

write

public static Managed.ManagedTransform write(java.lang.String sink)

Instantiates a Managed.ManagedTransform transform for the specified sink. The supported managed sinks are:

• ICEBERG : Write to Apache Iceberg tables IcebergIO
• KAFKA : Write to Apache Kafka topics KafkaIO
• BIGQUERY : Write to GCP BigQuery tables BigQueryIO