org.apache.beam.sdk.schemas.utils

Class AvroUtils

java.lang.Object

org.apache.beam.sdk.schemas.utils.AvroUtils

@Experimental(value=SCHEMAS)
public class AvroUtils
extends java.lang.Object

Utils to convert AVRO records to Beam rows.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	AvroUtils.AvroConvertType
static class	AvroUtils.AvroConvertValueForGetter
static class	AvroUtils.AvroConvertValueForSetter
static class	AvroUtils.FixedBytesField Wrapper for fixed byte fields.

Method Summary

Modifier and Type	Method and Description
static java.lang.Object	convertAvroFieldStrict(java.lang.Object value, Schema avroSchema, Schema.FieldType fieldType) Strict conversion from AVRO to Beam, strict because it doesn't do widening or narrowing during conversion.
static <T> SchemaUserTypeCreator	getCreator(java.lang.Class<T> clazz, Schema schema) Get an object creator for an AVRO-generated SpecificRecord.
static <T> java.util.List<FieldValueTypeInformation>	getFieldTypes(java.lang.Class<T> clazz, Schema schema) Get field types for an AVRO-generated SpecificRecord or a POJO.
static <T> SerializableFunction<Row,T>	getFromRowFunction(java.lang.Class<T> clazz)
static SerializableFunction<GenericRecord,Row>	getGenericRecordToRowFunction(Schema schema) Returns a function mapping AVRO GenericRecords to Beam Rows for use in PCollection.setSchema(org.apache.beam.sdk.schemas.Schema, org.apache.beam.sdk.values.TypeDescriptor<T>, org.apache.beam.sdk.transforms.SerializableFunction<T, org.apache.beam.sdk.values.Row>, org.apache.beam.sdk.transforms.SerializableFunction<org.apache.beam.sdk.values.Row, T>).
static <T> java.util.List<FieldValueGetter>	getGetters(java.lang.Class<T> clazz, Schema schema) Get generated getters for an AVRO-generated SpecificRecord or a POJO.
static SerializableFunction<Row,GenericRecord>	getRowToGenericRecordFunction(Schema avroSchema) Returns a function mapping Beam Rows to AVRO GenericRecords for use in PCollection.setSchema(org.apache.beam.sdk.schemas.Schema, org.apache.beam.sdk.values.TypeDescriptor<T>, org.apache.beam.sdk.transforms.SerializableFunction<T, org.apache.beam.sdk.values.Row>, org.apache.beam.sdk.transforms.SerializableFunction<org.apache.beam.sdk.values.Row, T>).
static <T> Schema	getSchema(java.lang.Class<T> clazz, Schema schema)
static <T> SerializableFunction<T,Row>	getToRowFunction(java.lang.Class<T> clazz, Schema schema)
static <T> SchemaCoder<T>	schemaCoder(AvroCoder<T> avroCoder) Returns an SchemaCoder instance based on the provided AvroCoder for the element type.
static <T> SchemaCoder<T>	schemaCoder(java.lang.Class<T> clazz) Returns an SchemaCoder instance for the provided element class.
static <T> SchemaCoder<T>	schemaCoder(java.lang.Class<T> clazz, Schema schema) Returns an SchemaCoder instance for the provided element type using the provided Avro schema.
static SchemaCoder<GenericRecord>	schemaCoder(Schema schema) Returns an SchemaCoder instance for the Avro schema.
static <T> SchemaCoder<T>	schemaCoder(TypeDescriptor<T> type) Returns an SchemaCoder instance for the provided element type.
static Schema.Field	toAvroField(Schema.Field field, java.lang.String namespace) Get Avro Field from Beam Field.
static Schema	toAvroSchema(Schema beamSchema)
static Schema	toAvroSchema(Schema beamSchema, java.lang.String name, java.lang.String namespace) Converts a Beam Schema into an AVRO schema.
static Schema.Field	toBeamField(Schema.Field field) Get Beam Field from avro Field.
static Row	toBeamRowStrict(GenericRecord record, Schema schema) Strict conversion from AVRO to Beam, strict because it doesn't do widening or narrowing during conversion.
static Schema	toBeamSchema(Schema schema) Converts AVRO schema to Beam row schema.
static GenericRecord	toGenericRecord(Row row, Schema avroSchema) Convert from a Beam Row to an AVRO GenericRecord.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

toBeamField

public static Schema.Field toBeamField(Schema.Field field)

Get Beam Field from avro Field.

toAvroField

public static Schema.Field toAvroField(Schema.Field field,
                                       java.lang.String namespace)

Get Avro Field from Beam Field.

toBeamSchema

public static Schema toBeamSchema(Schema schema)

Converts AVRO schema to Beam row schema.

Parameters:

schema - schema of type RECORD

toAvroSchema

public static Schema toAvroSchema(Schema beamSchema,
                                  @Nullable
                                  java.lang.String name,
                                  @Nullable
                                  java.lang.String namespace)

Converts a Beam Schema into an AVRO schema.

toAvroSchema

public static Schema toAvroSchema(Schema beamSchema)

toBeamRowStrict

public static Row toBeamRowStrict(GenericRecord record,
                                  @Nullable
                                  Schema schema)

Strict conversion from AVRO to Beam, strict because it doesn't do widening or narrowing during conversion. If Schema is not provided, one is inferred from the AVRO schema.

toGenericRecord

public static GenericRecord toGenericRecord(Row row,
                                            @Nullable
                                            Schema avroSchema)

Convert from a Beam Row to an AVRO GenericRecord. If a Schema is not provided, one is inferred from the Beam schema on the row.

getToRowFunction

public static <T> SerializableFunction<T,Row> getToRowFunction(java.lang.Class<T> clazz,
                                                               @Nullable
                                                               Schema schema)

getFromRowFunction

public static <T> SerializableFunction<Row,T> getFromRowFunction(java.lang.Class<T> clazz)

getSchema

@Nullable
public static <T> Schema getSchema(java.lang.Class<T> clazz,
                                             @Nullable
                                             Schema schema)

getGenericRecordToRowFunction

public static SerializableFunction<GenericRecord,Row> getGenericRecordToRowFunction(@Nullable
                                                                                    Schema schema)

Returns a function mapping AVRO GenericRecords to Beam Rows for use in PCollection.setSchema(org.apache.beam.sdk.schemas.Schema, org.apache.beam.sdk.values.TypeDescriptor<T>, org.apache.beam.sdk.transforms.SerializableFunction<T, org.apache.beam.sdk.values.Row>, org.apache.beam.sdk.transforms.SerializableFunction<org.apache.beam.sdk.values.Row, T>).

getRowToGenericRecordFunction

public static SerializableFunction<Row,GenericRecord> getRowToGenericRecordFunction(@Nullable
                                                                                    Schema avroSchema)

Returns a function mapping Beam Rows to AVRO GenericRecords for use in PCollection.setSchema(org.apache.beam.sdk.schemas.Schema, org.apache.beam.sdk.values.TypeDescriptor<T>, org.apache.beam.sdk.transforms.SerializableFunction<T, org.apache.beam.sdk.values.Row>, org.apache.beam.sdk.transforms.SerializableFunction<org.apache.beam.sdk.values.Row, T>).

schemaCoder

public static <T> SchemaCoder<T> schemaCoder(TypeDescriptor<T> type)

Returns an SchemaCoder instance for the provided element type.

Type Parameters:

T - the element type

schemaCoder

public static <T> SchemaCoder<T> schemaCoder(java.lang.Class<T> clazz)

Returns an SchemaCoder instance for the provided element class.

Type Parameters:

T - the element type

schemaCoder

public static SchemaCoder<GenericRecord> schemaCoder(Schema schema)

Returns an SchemaCoder instance for the Avro schema. The implicit type is GenericRecord.

schemaCoder

public static <T> SchemaCoder<T> schemaCoder(java.lang.Class<T> clazz,
                                             Schema schema)

Returns an SchemaCoder instance for the provided element type using the provided Avro schema.

If the type argument is GenericRecord, the schema may be arbitrary. Otherwise, the schema must correspond to the type provided.

Type Parameters:

T - the element type

schemaCoder

public static <T> SchemaCoder<T> schemaCoder(AvroCoder<T> avroCoder)

Returns an SchemaCoder instance based on the provided AvroCoder for the element type.

Type Parameters:

T - the element type

getFieldTypes

public static <T> java.util.List<FieldValueTypeInformation> getFieldTypes(java.lang.Class<T> clazz,
                                                                          Schema schema)

Get field types for an AVRO-generated SpecificRecord or a POJO.

getGetters

public static <T> java.util.List<FieldValueGetter> getGetters(java.lang.Class<T> clazz,
                                                              Schema schema)

Get generated getters for an AVRO-generated SpecificRecord or a POJO.

getCreator

public static <T> SchemaUserTypeCreator getCreator(java.lang.Class<T> clazz,
                                                   Schema schema)

Get an object creator for an AVRO-generated SpecificRecord.

convertAvroFieldStrict

@Nullable
public static java.lang.Object convertAvroFieldStrict(@Nullable
                                                                java.lang.Object value,
                                                                @Nonnull
                                                                Schema avroSchema,
                                                                @Nonnull
                                                                Schema.FieldType fieldType)

Strict conversion from AVRO to Beam, strict because it doesn't do widening or narrowing during conversion.

Parameters:

value - GenericRecord or any nested value

avroSchema - schema for value

fieldType - target beam field type

Returns:

value converted for Row