org.apache.beam.sdk.schemas.utils

Class AvroUtils

java.lang.Object

org.apache.beam.sdk.schemas.utils.AvroUtils

@Experimental(value=SCHEMAS)
public class AvroUtils
extends java.lang.Object

Utils to convert AVRO records to Beam rows.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	AvroUtils.FixedBytesField Wrapper for fixed byte fields.

Method Summary

Modifier and Type	Method and Description
static java.lang.Object	convertAvroFieldStrict(java.lang.Object value, Schema avroSchema, Schema.FieldType fieldType) Strict conversion from AVRO to Beam, strict because it doesn't do widening or narrowing during conversion.
static <T> SchemaUserTypeCreator	getCreator(java.lang.Class<T> clazz, Schema schema) Get an object creator for an AVRO-generated SpecificRecord.
static <T> java.util.List<FieldValueTypeInformation>	getFieldTypes(java.lang.Class<T> clazz, Schema schema) Get field types for an AVRO-generated SpecificRecord or a POJO.
static <T> SerializableFunction<Row,T>	getFromRowFunction(java.lang.Class<T> clazz)
static SerializableFunction<GenericRecord,Row>	getGenericRecordToRowFunction(Schema schema) Returns a function mapping AVRO GenericRecords to Beam Rows for use in PCollection.setSchema(org.apache.beam.sdk.schemas.Schema, org.apache.beam.sdk.transforms.SerializableFunction<T, org.apache.beam.sdk.values.Row>, org.apache.beam.sdk.transforms.SerializableFunction<org.apache.beam.sdk.values.Row, T>).
static <T> java.util.List<FieldValueGetter>	getGetters(java.lang.Class<T> clazz, Schema schema) Get generated getters for an AVRO-generated SpecificRecord or a POJO.
static SerializableFunction<Row,GenericRecord>	getRowToGenericRecordFunction(Schema avroSchema) Returns a function mapping Beam Rows to AVRO GenericRecords for use in PCollection.setSchema(org.apache.beam.sdk.schemas.Schema, org.apache.beam.sdk.transforms.SerializableFunction<T, org.apache.beam.sdk.values.Row>, org.apache.beam.sdk.transforms.SerializableFunction<org.apache.beam.sdk.values.Row, T>).
static <T> Schema	getSchema(java.lang.Class<T> clazz, Schema schema)
static <T> SerializableFunction<T,Row>	getToRowFunction(java.lang.Class<T> clazz, Schema schema)
static Schema.Field	toAvroField(Schema.Field field) Get Avro Field from Beam Field.
static Schema	toAvroSchema(Schema beamSchema) Converts a Beam Schema into an AVRO schema.
static Schema.Field	toBeamField(Schema.Field field) Get Beam Field from avro Field.
static Row	toBeamRowStrict(GenericRecord record, Schema schema) Strict conversion from AVRO to Beam, strict because it doesn't do widening or narrowing during conversion.
static Schema	toBeamSchema(Schema schema) Converts AVRO schema to Beam row schema.
static GenericRecord	toGenericRecord(Row row, Schema avroSchema) Convert from a Beam Row to an AVRO GenericRecord.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

toBeamField

public static Schema.Field toBeamField(Schema.Field field)

Get Beam Field from avro Field.

toAvroField

public static Schema.Field toAvroField(Schema.Field field)

Get Avro Field from Beam Field.

toBeamSchema

public static Schema toBeamSchema(Schema schema)

Converts AVRO schema to Beam row schema.

Parameters:

schema - schema of type RECORD

toAvroSchema

public static Schema toAvroSchema(Schema beamSchema)

Converts a Beam Schema into an AVRO schema.

toBeamRowStrict

public static Row toBeamRowStrict(GenericRecord record,
                                  @Nullable
                                  Schema schema)

Strict conversion from AVRO to Beam, strict because it doesn't do widening or narrowing during conversion. If Schema is not provided, one is inferred from the AVRO schema.

toGenericRecord

public static GenericRecord toGenericRecord(Row row,
                                            @Nullable
                                            Schema avroSchema)

Convert from a Beam Row to an AVRO GenericRecord. If a Schema is not provided, one is inferred from the Beam schema on the orw.

getToRowFunction

public static <T> SerializableFunction<T,Row> getToRowFunction(java.lang.Class<T> clazz,
                                                               @Nullable
                                                               Schema schema)

getFromRowFunction

public static <T> SerializableFunction<Row,T> getFromRowFunction(java.lang.Class<T> clazz)

getSchema

@Nullable
public static <T> Schema getSchema(java.lang.Class<T> clazz,
                                             @Nullable
                                             Schema schema)

getGenericRecordToRowFunction

public static SerializableFunction<GenericRecord,Row> getGenericRecordToRowFunction(@Nullable
                                                                                    Schema schema)

Returns a function mapping AVRO GenericRecords to Beam Rows for use in PCollection.setSchema(org.apache.beam.sdk.schemas.Schema, org.apache.beam.sdk.transforms.SerializableFunction<T, org.apache.beam.sdk.values.Row>, org.apache.beam.sdk.transforms.SerializableFunction<org.apache.beam.sdk.values.Row, T>).

getRowToGenericRecordFunction

public static SerializableFunction<Row,GenericRecord> getRowToGenericRecordFunction(@Nullable
                                                                                    Schema avroSchema)

Returns a function mapping Beam Rows to AVRO GenericRecords for use in PCollection.setSchema(org.apache.beam.sdk.schemas.Schema, org.apache.beam.sdk.transforms.SerializableFunction<T, org.apache.beam.sdk.values.Row>, org.apache.beam.sdk.transforms.SerializableFunction<org.apache.beam.sdk.values.Row, T>).

getFieldTypes

public static <T> java.util.List<FieldValueTypeInformation> getFieldTypes(java.lang.Class<T> clazz,
                                                                          Schema schema)

Get field types for an AVRO-generated SpecificRecord or a POJO.

getGetters

public static <T> java.util.List<FieldValueGetter> getGetters(java.lang.Class<T> clazz,
                                                              Schema schema)

Get generated getters for an AVRO-generated SpecificRecord or a POJO.

getCreator

public static <T> SchemaUserTypeCreator getCreator(java.lang.Class<T> clazz,
                                                   Schema schema)

Get an object creator for an AVRO-generated SpecificRecord.

convertAvroFieldStrict

@Nullable
public static java.lang.Object convertAvroFieldStrict(@Nullable
                                                                java.lang.Object value,
                                                                @Nonnull
                                                                Schema avroSchema,
                                                                @Nonnull
                                                                Schema.FieldType fieldType)

Strict conversion from AVRO to Beam, strict because it doesn't do widening or narrowing during conversion.

Parameters:

value - GenericRecord or any nested value

avroSchema - schema for value

fieldType - target beam field type

Returns:

value converted for Row