java.lang.Object

org.apache.beam.sdk.extensions.avro.schemas.utils.AvroUtils

public class AvroUtils extends Object

Utils to convert AVRO records to Beam rows. Imposes a mapping between common avro types and Beam portable schemas (https://s.apache.org/beam-schemas):

   Avro                Beam Field Type
   INT         invalid input: '<'-----> INT32
   LONG        invalid input: '<'-----> INT64
   FLOAT       invalid input: '<'-----> FLOAT
   DOUBLE      invalid input: '<'-----> DOUBLE
   BOOLEAN     invalid input: '<'-----> BOOLEAN
   STRING      invalid input: '<'-----> STRING
   BYTES       invalid input: '<'-----> BYTES
               invalid input: '<'------ LogicalType(urn="beam:logical_type:var_bytes:v1")
   FIXED       invalid input: '<'-----> LogicalType(urn="beam:logical_type:fixed_bytes:v1")
   ARRAY       invalid input: '<'-----> ARRAY
   ENUM        invalid input: '<'-----> LogicalType(EnumerationType)
   MAP         invalid input: '<'-----> MAP
   RECORD      invalid input: '<'-----> ROW
   UNION       invalid input: '<'-----> LogicalType(OneOfType)
   LogicalTypes.Date              invalid input: '<'-----> LogicalType(DATE)
                                  invalid input: '<'------ LogicalType(urn="beam:logical_type:date:v1")
   LogicalTypes.TimestampMillis   invalid input: '<'-----> DATETIME
   LogicalTypes.Decimal           invalid input: '<'-----> DECIMAL

For SQL CHAR/VARCHAR types, an Avro schema

   LogicalType({"type":"string","logicalType":"char","maxLength":MAX_LENGTH}) or
   LogicalType({"type":"string","logicalType":"varchar","maxLength":MAX_LENGTH})

is used.

Nested Class Summary

Nested Classes

Modifier and Type

Class

Description

static class

AvroUtils.AvroConvertType

static class

AvroUtils.AvroConvertValueForGetter

static class

AvroUtils.AvroConvertValueForSetter

static class

AvroUtils.FixedBytesField

Wrapper for fixed byte fields.

static class

AvroUtils.TypeWithNullability
Method Summary

Modifier and Type

Method

Description

static void

addLogicalTypeConversions(GenericData data)

static @PolyNull Object

convertAvroFieldStrict(@PolyNull Object value, Schema avroSchema, Schema.FieldType fieldType)

Strict conversion from AVRO to Beam, strict because it doesn't do widening or narrowing during conversion.

static @PolyNull Object

convertAvroFieldStrict(@PolyNull Object value, Schema avroSchema, Schema.FieldType fieldType, GenericData genericData)

Strict conversion from AVRO to Beam, strict because it doesn't do widening or narrowing during conversion.

static SimpleFunction<byte[],Row>

getAvroBytesToRowFunction(Schema beamSchema)

Returns a function mapping encoded AVRO GenericRecords to Beam Rows.

static <T> SchemaUserTypeCreator

getCreator(TypeDescriptor<T> typeDescriptor, Schema schema)

Get an object creator for an AVRO-generated SpecificRecord.

static <T> List<FieldValueTypeInformation>

getFieldTypes(TypeDescriptor<T> typeDescriptor, Schema schema)

Get field types for an AVRO-generated SpecificRecord or a POJO.

static <T> SerializableFunction<Row,T>

getFromRowFunction(Class<T> clazz)

static SerializableFunction<GenericRecord,Row>

getGenericRecordToRowFunction(@Nullable Schema schema)

Returns a function mapping AVRO GenericRecords to Beam Rows for use in PCollection.setSchema(org.apache.beam.sdk.schemas.Schema, org.apache.beam.sdk.values.TypeDescriptor<T>, org.apache.beam.sdk.transforms.SerializableFunction<T, org.apache.beam.sdk.values.Row>, org.apache.beam.sdk.transforms.SerializableFunction<org.apache.beam.sdk.values.Row, T>).

static <T> List<FieldValueGetter<@NonNull T,Object>>

getGetters(TypeDescriptor<T> typeDescriptor, Schema schema)

Get generated getters for an AVRO-generated SpecificRecord or a POJO.

static SimpleFunction<Row,byte[]>

getRowToAvroBytesFunction(Schema beamSchema)

Returns a function mapping Beam Rows to encoded AVRO GenericRecords.

static SerializableFunction<Row,GenericRecord>

getRowToGenericRecordFunction(@Nullable Schema avroSchema)

Returns a function mapping Beam Rows to AVRO GenericRecords for use in PCollection.setSchema(org.apache.beam.sdk.schemas.Schema, org.apache.beam.sdk.values.TypeDescriptor<T>, org.apache.beam.sdk.transforms.SerializableFunction<T, org.apache.beam.sdk.values.Row>, org.apache.beam.sdk.transforms.SerializableFunction<org.apache.beam.sdk.values.Row, T>).

static <T> @Nullable Schema

getSchema(Class<T> clazz, @Nullable Schema schema)

static <T> SerializableFunction<T,Row>

getToRowFunction(Class<T> clazz, Schema schema)

static <T> SchemaCoder<T>

schemaCoder(Class<T> clazz)

Returns an SchemaCoder instance for the provided element class.

static <T> SchemaCoder<T>

schemaCoder(Class<T> clazz, Schema schema)

Returns an SchemaCoder instance for the provided element type using the provided Avro schema.

static SchemaCoder<GenericRecord>

schemaCoder(Schema schema)

Returns an SchemaCoder instance for the Avro schema.

static <T> SchemaCoder<T>

schemaCoder(AvroCoder<T> avroCoder)

Returns an SchemaCoder instance based on the provided AvroCoder for the element type.

static <T> SchemaCoder<T>

schemaCoder(TypeDescriptor<T> type)

Returns an SchemaCoder instance for the provided element type.

static Schema.Field

toAvroField(Schema.Field field, String namespace)

Get Avro Field from Beam Field.

static Schema

toAvroSchema(Schema beamSchema)

static Schema

toAvroSchema(Schema beamSchema, @Nullable String name, @Nullable String namespace)

Converts a Beam Schema into an AVRO schema.

static Schema.Field

toBeamField(Schema.Field field)

Get Beam Field from avro Field.

static Row

toBeamRowStrict(GenericRecord record, @Nullable Schema schema)

Strict conversion from AVRO to Beam, strict because it doesn't do widening or narrowing during conversion.

static Row

toBeamRowStrict(GenericRecord record, @Nullable Schema schema, @Nullable GenericData genericData)

Strict conversion from AVRO to Beam, strict because it doesn't do widening or narrowing during conversion.

static Schema

toBeamSchema(Class<?> clazz)

Converts AVRO schema to Beam row schema.

static Schema

toBeamSchema(Schema schema)

Converts AVRO schema to Beam row schema.

static GenericRecord

toGenericRecord(Row row)

Convert from a Beam Row to an AVRO GenericRecord.

static GenericRecord

toGenericRecord(Row row, @Nullable Schema avroSchema)

Convert from a Beam Row to an AVRO GenericRecord.

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Details
- addLogicalTypeConversions
  
  public static void addLogicalTypeConversions(GenericData data)
- toBeamField
  
  public static Schema.Field toBeamField(Schema.Field field)
  
  Get Beam Field from avro Field.
- toAvroField
  
  public static Schema.Field toAvroField(Schema.Field field, String namespace)
  
  Get Avro Field from Beam Field.
- toBeamSchema
  
  public static Schema toBeamSchema(Class<?> clazz)
  
  Converts AVRO schema to Beam row schema.
  
  Parameters:
  
  clazz - avro class
- toBeamSchema
  
  public static Schema toBeamSchema(Schema schema)
  
  Converts AVRO schema to Beam row schema.
  
  Parameters:
  
  schema - schema of type RECORD
- toAvroSchema
  
  public static Schema toAvroSchema(Schema beamSchema, @Nullable String name, @Nullable String namespace)
  
  Converts a Beam Schema into an AVRO schema.
- toAvroSchema
  
  public static Schema toAvroSchema(Schema beamSchema)
- toBeamRowStrict
  
  public static Row toBeamRowStrict(GenericRecord record, @Nullable Schema schema, @Nullable GenericData genericData)
  
  Strict conversion from AVRO to Beam, strict because it doesn't do widening or narrowing during conversion. If Schema is not provided, one is inferred from the AVRO schema.
- toBeamRowStrict
  
  public static Row toBeamRowStrict(GenericRecord record, @Nullable Schema schema)
  
  Strict conversion from AVRO to Beam, strict because it doesn't do widening or narrowing during conversion. If Schema is not provided, one is inferred from the AVRO schema.
- toGenericRecord
  
  public static GenericRecord toGenericRecord(Row row)
  
  Convert from a Beam Row to an AVRO GenericRecord. The Avro Schema is inferred from the Beam schema on the row.
- toGenericRecord
  
  public static GenericRecord toGenericRecord(Row row, @Nullable Schema avroSchema)
  
  Convert from a Beam Row to an AVRO GenericRecord. If a Schema is not provided, one is inferred from the Beam schema on the row.
- getToRowFunction
  
  public static <T> SerializableFunction<T,Row> getToRowFunction(Class<T> clazz, Schema schema)
- getFromRowFunction
  
  public static <T> SerializableFunction<Row,T> getFromRowFunction(Class<T> clazz)
- getSchema
  
  public static <T> @Nullable Schema getSchema(Class<T> clazz, @Nullable Schema schema)
- getAvroBytesToRowFunction
  
  public static SimpleFunction<byte[],Row> getAvroBytesToRowFunction(Schema beamSchema)
  
  Returns a function mapping encoded AVRO GenericRecords to Beam Rows.
- getRowToAvroBytesFunction
  
  public static SimpleFunction<Row,byte[]> getRowToAvroBytesFunction(Schema beamSchema)
  
  Returns a function mapping Beam Rows to encoded AVRO GenericRecords.
- getGenericRecordToRowFunction
  
  public static SerializableFunction<GenericRecord,Row> getGenericRecordToRowFunction(@Nullable Schema schema)
  
  Returns a function mapping AVRO GenericRecords to Beam Rows for use in PCollection.setSchema(org.apache.beam.sdk.schemas.Schema, org.apache.beam.sdk.values.TypeDescriptor<T>, org.apache.beam.sdk.transforms.SerializableFunction<T, org.apache.beam.sdk.values.Row>, org.apache.beam.sdk.transforms.SerializableFunction<org.apache.beam.sdk.values.Row, T>).
- getRowToGenericRecordFunction
  
  public static SerializableFunction<Row,GenericRecord> getRowToGenericRecordFunction(@Nullable Schema avroSchema)
  
  Returns a function mapping Beam Rows to AVRO GenericRecords for use in PCollection.setSchema(org.apache.beam.sdk.schemas.Schema, org.apache.beam.sdk.values.TypeDescriptor<T>, org.apache.beam.sdk.transforms.SerializableFunction<T, org.apache.beam.sdk.values.Row>, org.apache.beam.sdk.transforms.SerializableFunction<org.apache.beam.sdk.values.Row, T>).
- schemaCoder
  
  public static <T> SchemaCoder<T> schemaCoder(TypeDescriptor<T> type)
  
  Returns an SchemaCoder instance for the provided element type.
  
  Type Parameters:
  
  T - the element type
- schemaCoder
  
  public static <T> SchemaCoder<T> schemaCoder(Class<T> clazz)
  
  Returns an SchemaCoder instance for the provided element class.
  
  Type Parameters:
  
  T - the element type
- schemaCoder
  
  public static SchemaCoder<GenericRecord> schemaCoder(Schema schema)
  
  Returns an SchemaCoder instance for the Avro schema. The implicit type is GenericRecord.
- schemaCoder
  
  public static <T> SchemaCoder<T> schemaCoder(Class<T> clazz, Schema schema)
  
  Returns an SchemaCoder instance for the provided element type using the provided Avro schema.
  If the type argument is GenericRecord, the schema may be arbitrary. Otherwise, the schema must correspond to the type provided.
  
  Type Parameters:
  
  T - the element type
- schemaCoder
  
  public static <T> SchemaCoder<T> schemaCoder(AvroCoder<T> avroCoder)
  
  Returns an SchemaCoder instance based on the provided AvroCoder for the element type.
  
  Type Parameters:
  
  T - the element type
- getFieldTypes
  
  public static <T> List<FieldValueTypeInformation> getFieldTypes(TypeDescriptor<T> typeDescriptor, Schema schema)
  
  Get field types for an AVRO-generated SpecificRecord or a POJO.
- getGetters
  
  public static <T> List<FieldValueGetter<@NonNull T,Object>> getGetters(TypeDescriptor<T> typeDescriptor, Schema schema)
  
  Get generated getters for an AVRO-generated SpecificRecord or a POJO.
- getCreator
  
  public static <T> SchemaUserTypeCreator getCreator(TypeDescriptor<T> typeDescriptor, Schema schema)
  
  Get an object creator for an AVRO-generated SpecificRecord.
- convertAvroFieldStrict
  
  public static @PolyNull Object convertAvroFieldStrict(@PolyNull Object value, @Nonnull Schema avroSchema, @Nonnull Schema.FieldType fieldType, @Nonnull GenericData genericData)
  
  Strict conversion from AVRO to Beam, strict because it doesn't do widening or narrowing during conversion.
  
  Parameters:
  
  value - GenericRecord or any nested value
  
  avroSchema - schema for value
  
  fieldType - target beam field type
  
  genericData - GenericData instance to use for conversions
  
  Returns:
  
  value converted for Row
- convertAvroFieldStrict
  
  public static @PolyNull Object convertAvroFieldStrict(@PolyNull Object value, @Nonnull Schema avroSchema, @Nonnull Schema.FieldType fieldType)
  
  Strict conversion from AVRO to Beam, strict because it doesn't do widening or narrowing during conversion.
  
  Parameters:
  
  value - GenericRecord or any nested value
  
  avroSchema - schema for value
  
  fieldType - target beam field type
  
  Returns:
  
  value converted for Row

Class AvroUtils

Nested Class Summary

Method Summary

Methods inherited from class java.lang.Object

Method Details

addLogicalTypeConversions

toBeamField

toAvroField

toBeamSchema

toBeamSchema

toAvroSchema

toAvroSchema

toBeamRowStrict

toBeamRowStrict

toGenericRecord

toGenericRecord

getToRowFunction

getFromRowFunction

getSchema

getAvroBytesToRowFunction

getRowToAvroBytesFunction

getGenericRecordToRowFunction

getRowToGenericRecordFunction

schemaCoder

schemaCoder

schemaCoder

schemaCoder

schemaCoder

getFieldTypes

getGetters

getCreator

convertAvroFieldStrict

convertAvroFieldStrict